Oct 30 03:00:03.475508 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 03:00:03.475519 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 03:00:03.475529 ip-10-0-136-97 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 03:00:03.475855 ip-10-0-136-97 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 03:00:13.724255 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 03:00:13.724271 ip-10-0-136-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5263febe3f7e4869839a9a45d3c75447 -- Oct 30 03:02:09.868652 ip-10-0-136-97 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:02:10.301399 ip-10-0-136-97 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:10.301399 ip-10-0-136-97 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:02:10.301399 ip-10-0-136-97 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:10.301399 ip-10-0-136-97 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:10.305897 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305815 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.305897 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305897 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305901 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305905 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305908 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305910 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305913 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305916 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305918 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305921 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305924 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305927 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305929 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305933 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305937 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305940 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305943 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305945 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305948 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305951 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.305967 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305953 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305956 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305958 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305961 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305964 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305966 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305969 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305979 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305982 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305985 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305988 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305990 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305993 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305996 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.305999 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306002 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306004 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306007 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306009 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306012 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.306490 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306014 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306017 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306019 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306022 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306024 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306027 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306030 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306034 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306037 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306040 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306043 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306045 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306048 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306050 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306054 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306057 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306060 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306063 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306066 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.306990 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306070 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306072 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306075 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306077 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306080 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306084 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306086 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306089 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306092 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306094 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306097 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306099 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306102 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306105 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306107 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306110 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306113 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306115 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306118 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.307457 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306121 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306123 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306126 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306128 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306131 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306135 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306138 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306140 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306145 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306151 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306154 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306157 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306160 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306162 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306165 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306168 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306171 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306174 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306176 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.307894 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306180 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306182 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306185 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306188 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306190 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306193 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306196 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306198 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.306201 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307031 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307039 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307042 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307045 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307048 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307051 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307053 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307058 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307061 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307063 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307066 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.308360 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307069 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307073 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307077 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307080 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307082 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307085 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307088 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307091 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307094 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307096 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307099 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307102 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307104 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307107 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307111 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307115 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307118 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307121 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307124 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.308875 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307127 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307130 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307132 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307135 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307138 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307141 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307143 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307146 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307149 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307152 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307155 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307157 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307160 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307162 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307165 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307167 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307170 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307172 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307175 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307178 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.309329 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307181 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307184 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307187 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307189 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307192 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307195 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307198 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307202 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307206 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307210 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307213 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307230 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307234 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307238 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307241 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307243 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307246 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307249 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307252 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.309809 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307255 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307258 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307260 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307263 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307265 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307268 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307270 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307273 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307275 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307278 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307281 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307283 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307286 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307288 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307291 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307294 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307297 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307299 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307302 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307306 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.310287 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307308 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307311 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307313 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307316 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307318 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307321 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307324 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307327 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307329 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307332 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307334 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307337 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307340 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307343 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307346 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307348 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307351 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307879 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307885 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307888 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.310772 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307892 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307895 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307899 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307902 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307904 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307907 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307912 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307916 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307920 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307923 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307926 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307928 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307931 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307934 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307936 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307939 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307941 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307944 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307947 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.311234 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307949 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307952 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307954 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307957 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307962 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307964 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307967 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307970 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307972 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307975 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307979 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307981 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307984 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307986 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307989 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307991 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307994 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307996 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.307999 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308002 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.311695 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308005 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308008 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308010 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308013 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308015 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308018 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308021 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308023 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308026 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308028 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308031 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308034 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308036 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308039 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308041 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308044 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308047 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308057 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308060 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308062 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.312166 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308065 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308068 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308070 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308073 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308076 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308079 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308082 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308085 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308087 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308091 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308096 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308099 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308102 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308105 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308108 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308111 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308113 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308116 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308118 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.312648 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308121 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308123 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308126 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308129 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308131 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308134 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308136 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308139 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308142 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308146 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308149 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308151 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308154 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308157 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308159 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308162 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308164 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308168 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308171 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308173 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.313084 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308176 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308178 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308180 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308183 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.308186 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309338 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309347 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309355 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309360 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309375 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309379 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309384 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309388 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309392 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309395 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309399 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309402 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309405 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309409 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309412 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309415 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309419 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309422 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:02:10.313585 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309425 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309429 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309432 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309435 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309438 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309442 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309445 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309449 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309453 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309456 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309459 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309462 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309465 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309468 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309472 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309475 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309478 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309482 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309485 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309488 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309492 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309496 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309498 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309502 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:02:10.314114 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309504 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309510 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309513 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309516 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309521 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309524 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309527 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309531 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309534 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309537 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309540 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309543 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309548 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309551 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309554 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309557 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309561 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309564 2581 flags.go:64] FLAG: --help="false" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309567 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309570 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309573 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309576 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309580 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:02:10.314744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309583 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309587 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309590 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309596 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309599 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309602 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309605 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309608 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309611 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309614 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309617 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309620 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309623 2581 flags.go:64] FLAG: --lock-file="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309627 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309630 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309633 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309639 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309642 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309645 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309648 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309651 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309654 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309657 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309660 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309665 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:02:10.315277 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309668 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309672 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309675 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309679 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309681 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309685 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309688 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309691 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309694 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309700 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309704 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309708 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309711 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309714 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309717 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309720 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309723 2581 flags.go:64] FLAG: --port="10250" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309726 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309729 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00bd38562a7dd02fe" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309732 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309736 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309739 2581 flags.go:64] FLAG: --register-node="true" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309742 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309746 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309749 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309752 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:02:10.315894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309755 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309759 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309762 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309765 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309768 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309771 2581 flags.go:64] FLAG: --runonce="false" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309773 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309777 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309780 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309783 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309786 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309788 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309791 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309795 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309798 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309800 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309804 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309807 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309810 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309814 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309817 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309823 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309825 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309828 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309832 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:02:10.316583 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309835 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309839 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309842 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309846 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309849 2581 flags.go:64] FLAG: --v="2" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309854 2581 flags.go:64] FLAG: --version="false" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309858 2581 flags.go:64] FLAG: --vmodule="" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309862 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.309866 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310013 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310018 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310021 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310024 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310027 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310030 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310033 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310035 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310039 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310043 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310046 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310049 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310051 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.317189 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310054 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310056 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310059 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310062 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310065 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310068 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310070 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310073 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310076 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310078 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310081 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310085 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310088 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310092 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310095 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310097 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310100 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310102 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310105 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.317740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310108 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310110 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310113 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310115 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310118 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310121 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310123 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310126 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310128 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310131 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310133 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310136 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310139 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310141 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310144 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310147 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310150 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310152 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310155 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.318187 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310161 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310163 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310166 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310169 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310172 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310175 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310179 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310182 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310184 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310187 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310189 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310192 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310195 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310197 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310200 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310203 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310206 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310208 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310211 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310214 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.318706 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310217 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310219 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310222 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310225 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310227 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310230 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310233 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310235 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310238 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310241 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310243 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310246 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310249 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310252 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310254 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310257 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310261 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310263 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.319173 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310267 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310271 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310274 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310277 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310280 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310283 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310285 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310288 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310291 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310294 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310296 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310299 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310302 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310304 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310307 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310310 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.319758 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.310335 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310391 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310395 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310398 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310400 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310403 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310406 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310409 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310411 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310414 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310417 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310420 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310423 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310425 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310428 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310431 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310434 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310438 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310440 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310443 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.320144 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310446 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310448 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310451 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310454 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310456 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310459 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310463 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310467 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310470 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310473 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310477 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310479 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310482 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310485 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310487 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310490 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310492 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310495 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310497 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.320628 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310500 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310503 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310506 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310508 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310511 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310514 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310516 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310519 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310522 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310524 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310529 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310533 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310536 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310539 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310542 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310544 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310547 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310550 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310552 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310555 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.321062 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310558 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310561 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310563 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310566 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310568 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310571 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310573 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310576 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310578 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310581 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310584 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310586 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310589 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310592 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310595 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310597 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310600 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310603 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310606 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.321595 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310609 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310611 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310614 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310618 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310620 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310623 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310625 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310628 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310630 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310633 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310636 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310639 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310642 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310644 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310647 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310650 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310652 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310655 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310657 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310660 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.322038 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310663 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310665 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310668 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310670 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310673 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310676 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310678 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310681 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.310683 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.318613 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.318629 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.319394 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.319400 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319480 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319485 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319489 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319492 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319495 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319499 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.322519 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319501 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319504 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319507 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319510 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319513 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319522 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319524 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319528 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319530 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319533 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319536 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319539 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319541 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319544 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319547 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319550 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319552 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319555 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319558 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319560 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.322978 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319563 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319566 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319569 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319572 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319575 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319578 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319581 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319584 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319589 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319592 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319594 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319597 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319600 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319603 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319605 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319608 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319611 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319614 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319622 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.323465 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319625 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319628 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319630 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319633 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319636 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319638 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319641 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319643 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319646 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319649 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319651 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319654 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319657 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319659 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319662 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319665 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319668 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319671 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319674 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319677 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.323903 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319679 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319682 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319684 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319687 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319690 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319695 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319698 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319703 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319706 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319708 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319718 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319721 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319724 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319726 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319729 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319731 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319734 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.324417 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319737 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319739 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319742 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319750 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319753 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319756 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319758 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319761 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319764 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319767 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319770 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319773 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319775 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319779 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319781 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319784 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319786 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319789 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319793 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.325119 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319797 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319800 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.319802 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.319831 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320036 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320042 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320045 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320054 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320057 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320059 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320062 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320065 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320068 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320071 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320073 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320076 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.325855 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320079 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320081 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320084 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320087 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320089 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320092 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320095 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320098 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320101 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320103 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320106 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320109 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320113 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320117 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320120 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320123 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320125 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320128 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320131 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.326245 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320134 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320136 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320139 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320142 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320145 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320154 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320158 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320160 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320163 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320165 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320168 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320171 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320173 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320176 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320178 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320181 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320183 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320186 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320189 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320191 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.326723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320194 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320196 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320199 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320201 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320204 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320207 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320209 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320212 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320214 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320217 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320219 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320222 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320225 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320227 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320230 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320233 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320235 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320238 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320246 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.327253 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320249 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320251 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320254 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320258 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320261 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320264 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320267 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320270 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320272 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320275 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320277 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320280 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320283 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320285 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320288 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320290 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320293 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320296 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320298 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.327844 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320301 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320304 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320307 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320309 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320312 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320314 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320317 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320319 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320322 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320327 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320329 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320332 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320335 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320343 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320346 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320348 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.328276 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320351 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.320404 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320451 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320455 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320458 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320461 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320464 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320467 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320470 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320472 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320475 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320478 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320480 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320483 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320485 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320488 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:10.328665 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320490 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320493 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320496 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320499 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320501 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320504 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320507 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320510 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320512 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320518 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320522 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320525 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320528 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320531 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320540 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320543 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320546 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320549 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320551 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:10.329029 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320554 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320556 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320559 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320561 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320564 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320567 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320569 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320572 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320575 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320577 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320580 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320582 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320585 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320587 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320590 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320593 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320596 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320598 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320601 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320603 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:10.329566 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320606 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320608 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320612 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320615 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320617 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320620 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320622 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320625 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320633 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320636 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320639 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320642 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320644 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320647 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320649 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320652 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320655 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320657 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320661 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:10.330018 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320664 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320668 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320672 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320675 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320677 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320680 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320682 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320688 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320690 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320693 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320696 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320699 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320702 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320704 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320707 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320711 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320713 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320716 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320719 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320721 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:10.330479 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320724 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320727 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320736 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320739 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320742 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320744 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320753 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320756 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320758 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320761 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320764 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320766 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320769 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:10.320772 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.320833 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.324268 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.328239 2581 server.go:1014] "Starting client certificate rotation" Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.328337 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:10.330961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.328399 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:10.331428 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.330769 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:10.333990 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.333976 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:10.348984 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.348966 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:02:10.352886 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.352869 2581 log.go:25] "Validated CRI v1 image API" Oct 30 03:02:10.353815 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.353777 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:02:10.356470 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.356444 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d5b97899-7b6e-4f0a-acb4-9d1e441f77fa:/dev/nvme0n1p3 e7a15d70-85a0-48c1-b892-2e2942631c3e:/dev/nvme0n1p4] Oct 30 03:02:10.356545 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.356470 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 03:02:10.362546 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.362433 2581 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:10.361133953 +0000 UTC m=+0.377610190 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095659 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f2339cecd52a0852cdbaeffe9e09 SystemUUID:ec24f233-9cec-d52a-0852-cdbaeffe9e09 BootID:5263febe-3f7e-4869-839a-9a45d3c75447 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:66:74:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:66:74:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:b0:ad:56:10:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:02:10.362546 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.362546 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:02:10.362672 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.362627 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:02:10.363894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.363874 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:02:10.364035 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.363897 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:02:10.364076 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.364038 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:02:10.364076 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.364047 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:02:10.364128 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.364080 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:10.364956 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.364946 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:10.366192 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.366181 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:02:10.370854 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.370837 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:02:10.370966 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.370956 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:02:10.371826 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.371813 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:02:10.371899 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.371829 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:02:10.371899 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.371843 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:02:10.371899 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.371855 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:02:10.371899 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.371863 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:02:10.372227 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.372215 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:10.375939 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.375922 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:02:10.383764 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.383747 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:10.384602 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384590 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:02:10.384648 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384620 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:02:10.384771 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384760 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:02:10.384803 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384779 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:02:10.384803 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384787 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:02:10.384803 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384793 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:02:10.384803 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384799 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:02:10.384903 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.384805 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:02:10.385550 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385541 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:02:10.385588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385552 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:02:10.385588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385561 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:02:10.385588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385567 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:02:10.385588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385575 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:02:10.385588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.385584 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:02:10.386307 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.386298 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:02:10.386340 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.386309 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:02:10.390220 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.390207 2581 server.go:1263] "Started kubelet" Oct 30 03:02:10.390320 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.390290 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:02:10.390359 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.390303 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:02:10.390403 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.390383 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:02:10.390911 ip-10-0-136-97 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:02:10.391991 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.391973 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:02:10.393029 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.393014 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:02:10.397004 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.396987 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 03:02:10.399952 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.399930 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:10.400610 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.400596 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:02:10.400725 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.400706 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.401419 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401399 2581 factory.go:55] Registering systemd factory Oct 30 03:02:10.401419 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401416 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:02:10.401540 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401489 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:02:10.401540 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401505 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:02:10.401622 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401492 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:02:10.401660 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401654 2581 factory.go:153] Registering CRI-O factory Oct 30 03:02:10.401689 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401667 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 03:02:10.401689 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401684 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:02:10.401743 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401695 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:02:10.401743 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401712 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:02:10.401743 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401731 2581 factory.go:103] Registering Raw factory Oct 30 03:02:10.401832 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.401745 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 03:02:10.402750 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.402736 2581 manager.go:319] Starting recovery of all containers Oct 30 03:02:10.408885 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.408853 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:02:10.414107 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.414083 2581 manager.go:324] Recovery completed Oct 30 03:02:10.419546 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.419531 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.420646 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.420625 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:02:10.421345 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.421235 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 03:02:10.422445 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.421235 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-97.ec2.internal.18e32f8647322159 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-97.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-97.ec2.internal,},FirstTimestamp:2026-10-30 03:02:10.390180185 +0000 UTC m=+0.406656394,LastTimestamp:2026-10-30 03:02:10.390180185 +0000 UTC m=+0.406656394,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-97.ec2.internal,}" Oct 30 03:02:10.426506 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.426487 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.426570 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.426523 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.426570 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.426537 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.427140 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427116 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:02:10.427140 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427131 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.427220 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427132 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:02:10.427220 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427177 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:02:10.427813 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427798 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.427864 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427825 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.427864 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.427839 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.429354 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.429335 2581 policy_none.go:50] "Start" Oct 30 03:02:10.429354 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.429351 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:02:10.429506 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.429361 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:02:10.431430 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.431416 2581 policy_none.go:44] "Start" Oct 30 03:02:10.432670 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.432649 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 03:02:10.445915 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.445893 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 03:02:10.447518 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.447501 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsskq" Oct 30 03:02:10.456523 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.456498 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 03:02:10.456609 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.456476 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-97.ec2.internal.18e32f86495c6d82 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-97.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-97.ec2.internal,},FirstTimestamp:2026-10-30 03:02:10.426506626 +0000 UTC m=+0.442982845,LastTimestamp:2026-10-30 03:02:10.426506626 +0000 UTC m=+0.442982845,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-97.ec2.internal,}" Oct 30 03:02:10.456609 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.456558 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 03:02:10.457111 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.457095 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsskq" Oct 30 03:02:10.465777 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.465748 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:02:10.468451 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468433 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 03:02:10.468518 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.468459 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:02:10.468518 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468467 2581 server.go:87] "Starting device plugin registration server" Oct 30 03:02:10.468695 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468682 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:02:10.468736 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468697 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:02:10.468963 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468797 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:02:10.468963 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468877 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:02:10.468963 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.468885 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:02:10.469438 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.469417 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:02:10.469498 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.469471 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.492853 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.474471 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-97.ec2.internal.18e32f86495cc66e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-97.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-97.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-97.ec2.internal,},FirstTimestamp:2026-10-30 03:02:10.42652939 +0000 UTC m=+0.443005601,LastTimestamp:2026-10-30 03:02:10.42652939 +0000 UTC m=+0.443005601,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-97.ec2.internal,}" Oct 30 03:02:10.523238 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.523214 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:02:10.523338 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.523242 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:02:10.523338 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.523264 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:02:10.523338 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.523300 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 03:02:10.546214 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.546194 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:10.569631 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.569579 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.570477 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.570460 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.570576 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.570495 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.570576 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.570510 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.570576 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.570541 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.601499 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.601477 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.601499 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.601501 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-97.ec2.internal\": node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.624425 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.624391 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"] Oct 30 03:02:10.624533 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.624466 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.625332 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.625316 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.625441 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.625349 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.625441 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.625362 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.627624 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.627608 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.627753 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.627738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.627799 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.627767 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.628261 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628245 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.628261 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628252 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.628416 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628275 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.628416 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628292 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.628416 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628277 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.628416 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.628384 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.630730 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.630712 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.630803 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.630736 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:10.631413 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.631389 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:10.631413 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.631414 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:10.631526 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.631427 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:10.648634 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.648610 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.660134 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.660110 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-97.ec2.internal\" not found" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.662957 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.662938 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-97.ec2.internal\" not found" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.704037 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.704007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.704164 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.704050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.704164 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.704070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.749321 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.749292 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.805010 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.804975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.805179 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.805027 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.805179 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.805056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.805179 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.805084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.805179 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.805098 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.805179 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.805116 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.849545 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.849488 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.949890 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:10.949857 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:10.964069 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.964048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:10.966218 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:10.966188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:11.050159 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:11.049981 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.368715 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.368649 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.469110 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.469082 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.545735 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.545689 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 03:02:12.546188 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.545855 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 03:02:12.570001 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.569977 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.670428 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.670359 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.681117 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.681087 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:11 +0000 UTC" deadline="2028-07-27 14:37:32.597694644 +0000 UTC" Oct 30 03:02:12.681117 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.681112 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15275h35m19.916586027s" Oct 30 03:02:12.719051 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.719021 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:12.731903 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.731880 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:12.770863 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.770826 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.838340 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.838315 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:12.843045 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.843026 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:12.843182 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.843055 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:12.843182 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.843066 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:12.863936 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:12.863903 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ab5ae10a035469fb072566acf5cdc5.slice/crio-9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2 WatchSource:0}: Error finding container 9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2: Status 404 returned error can't find the container with id 9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2 Oct 30 03:02:12.864257 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:12.864239 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc90fa10f5e4f787096a878d9a85228.slice/crio-3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240 WatchSource:0}: Error finding container 3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240: Status 404 returned error can't find the container with id 3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240 Oct 30 03:02:12.869373 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.869360 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:02:12.871232 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:12.871214 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:12.881094 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.881077 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:12.919548 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.919499 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:12.943613 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.943565 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:12.944463 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.944451 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:12.962069 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:12.962051 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:13.591625 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.591316 2581 apiserver.go:52] "Watching apiserver" Oct 30 03:02:13.618474 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.618445 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:02:13.618783 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.618763 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-q6xf9","openshift-multus/multus-nv6kb","openshift-network-operator/iptables-alerter-bm2cm","openshift-ovn-kubernetes/ovnkube-node-wc8bj","openshift-cluster-node-tuning-operator/tuned-vfts7","openshift-multus/multus-additional-cni-plugins-kqbn2","openshift-multus/network-metrics-daemon-755m9","kube-system/konnectivity-agent-r95wv","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","openshift-network-diagnostics/network-check-target-jmvnr","kube-system/global-pull-secret-syncer-sgjx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng","openshift-dns/node-resolver-fv2ns"] Oct 30 03:02:13.619106 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.619089 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:13.621334 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.621306 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.623293 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.623273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:13.623389 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.623343 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:13.625457 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.625436 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.629207 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.627790 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.630455 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.630434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.630554 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.630501 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:13.632424 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632493 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632493 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632464 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632493 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632605 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632605 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632605 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632605 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632616 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.632921 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.632717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lr9l\" (UniqueName: \"kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.635042 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.635025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.636263 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.636243 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.636394 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.636379 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\"" Oct 30 03:02:13.636473 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.636460 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.639096 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.639076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.640292 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.640224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.642589 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.642573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.642677 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.642623 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:13.645496 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.645479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.648319 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.648300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.662982 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.662950 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:02:13.663086 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.663023 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlr5g" Oct 30 03:02:13.669021 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.669001 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlr5g" Oct 30 03:02:13.721323 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.721294 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:02:13.733189 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mgv\" (UniqueName: \"kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdt2f\" (UniqueName: \"kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733350 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w444f\" (UniqueName: \"kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733453 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgckr\" (UniqueName: \"kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.733780 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jkz\" (UniqueName: \"kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733767 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2679x\" (UniqueName: \"kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.734505 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.733984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lr9l\" (UniqueName: \"kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8w5\" (UniqueName: \"kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.735252 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734240 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734382 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t945\" (UniqueName: \"kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.735917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734558 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734573 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734790 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.734890 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735106 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735406 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735740 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735761 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735875 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735916 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.735938 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.736584 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.736004 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.737429 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.736043 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.737429 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.736769 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:02:13.737429 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.736797 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:02:13.738527 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738489 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:02:13.738666 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738650 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.738723 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738650 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.738723 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738678 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:02:13.738723 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738705 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738734 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738736 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738775 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738798 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738814 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738821 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.738860 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738860 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738897 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738861 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738913 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738717 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738899 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.738979 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.739009 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\"" Oct 30 03:02:13.739130 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.739037 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\"" Oct 30 03:02:13.739433 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.739252 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.739433 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.739269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.747052 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.747000 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240"} Oct 30 03:02:13.748783 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.748759 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2"} Oct 30 03:02:13.749960 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.749930 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\"" Oct 30 03:02:13.750071 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750002 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:02:13.750071 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750048 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:02:13.750071 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750066 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:02:13.750279 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750261 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:02:13.750340 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750279 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:02:13.750340 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750317 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\"" Oct 30 03:02:13.750432 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750355 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\"" Oct 30 03:02:13.750432 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750385 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:02:13.750432 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750402 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\"" Oct 30 03:02:13.750581 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.750442 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\"" Oct 30 03:02:13.757846 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.757822 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lr9l\" (UniqueName: \"kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.835085 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.835085 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835085 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w444f\" (UniqueName: \"kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgckr\" (UniqueName: \"kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835249 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.835345 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jkz\" (UniqueName: \"kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835491 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2679x\" (UniqueName: \"kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835595 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836058 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835678 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835799 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835926 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835954 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.835977 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836010 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836405 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836435 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.836580 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836481 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8w5\" (UniqueName: \"kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836693 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837073 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.836753 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.836809 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.336783812 +0000 UTC m=+3.135109243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837255 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.837275 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837311 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.837331 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.337312642 +0000 UTC m=+3.135638058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837334 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.836753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t945\" (UniqueName: \"kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837426 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837418 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837274 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837486 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837574 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.837961 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837606 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837670 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837947 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.837987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mgv\" (UniqueName: \"kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.838739 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdt2f\" (UniqueName: \"kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838210 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838295 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838342 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838370 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838777 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838785 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838848 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838852 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838898 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.839620 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.838984 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.839100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.840136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.840337 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.840406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.840358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.842700 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.842649 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.859744 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.859684 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t945\" (UniqueName: \"kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.859846 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.859748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jkz\" (UniqueName: \"kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:13.859846 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.859838 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:13.859916 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.859862 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:13.859916 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.859875 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:13.863313 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.863288 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mgv\" (UniqueName: \"kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.866083 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.866060 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8w5\" (UniqueName: \"kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:13.866186 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.866104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w444f\" (UniqueName: \"kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.866255 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.866221 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgckr\" (UniqueName: \"kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:13.866411 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:13.866389 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.36635695 +0000 UTC m=+3.164682373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:13.866616 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.866499 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2679x\" (UniqueName: \"kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.868131 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.868116 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdt2f\" (UniqueName: \"kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.936469 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.936435 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:13.947224 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.947192 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda252df28_aacb_4d80_a585_1f812bf77f0c.slice/crio-45e996f216ba395a4fc976fd54ac8ea8013e1ecc8de931521f741d8b6535eef1 WatchSource:0}: Error finding container 45e996f216ba395a4fc976fd54ac8ea8013e1ecc8de931521f741d8b6535eef1: Status 404 returned error can't find the container with id 45e996f216ba395a4fc976fd54ac8ea8013e1ecc8de931521f741d8b6535eef1 Oct 30 03:02:13.947965 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.947941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:13.957273 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.957246 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ab58b7_145a_4e99_b95d_31b89c2c29a0.slice/crio-cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73 WatchSource:0}: Error finding container cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73: Status 404 returned error can't find the container with id cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73 Oct 30 03:02:13.976917 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.976884 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:13.977886 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.977864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:13.980532 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.980497 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:13.985203 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.985178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:13.987274 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.987252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nv6kb" Oct 30 03:02:13.990073 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.990045 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9b309a_c095_4296_8c2c_18059f0785e2.slice/crio-bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76 WatchSource:0}: Error finding container bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76: Status 404 returned error can't find the container with id bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76 Oct 30 03:02:13.990723 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.990697 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bad7eb_bfe1_43a0_9c79_6a48f6fb57ea.slice/crio-5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f WatchSource:0}: Error finding container 5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f: Status 404 returned error can't find the container with id 5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f Oct 30 03:02:13.995081 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:13.995052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:13.995821 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.995796 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706f817b_2517_45e0_8fb5_75cdb54bfd8c.slice/crio-9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828 WatchSource:0}: Error finding container 9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828: Status 404 returned error can't find the container with id 9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828 Oct 30 03:02:13.999585 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:13.999561 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76844ce_1723_4f25_af95_a7bf084f3d7c.slice/crio-53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f WatchSource:0}: Error finding container 53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f: Status 404 returned error can't find the container with id 53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f Oct 30 03:02:14.001857 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.001839 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:14.003798 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.003776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:14.005740 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:14.005692 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7e1221_1e48_4518_8b72_f0d6876b9e95.slice/crio-b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4 WatchSource:0}: Error finding container b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4: Status 404 returned error can't find the container with id b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4 Oct 30 03:02:14.013345 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:14.013320 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eace382_e677_4a34_9e4e_8c1cd06d0eca.slice/crio-d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940 WatchSource:0}: Error finding container d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940: Status 404 returned error can't find the container with id d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940 Oct 30 03:02:14.016215 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:14.016169 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf7caae_56b4_4610_878e_3dd5a3dda185.slice/crio-40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07 WatchSource:0}: Error finding container 40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07: Status 404 returned error can't find the container with id 40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07 Oct 30 03:02:14.342458 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.342425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:14.342640 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.342473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:14.342705 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.342640 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:14.342756 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.342706 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:15.342689296 +0000 UTC m=+4.141014697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:14.343099 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.343081 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:14.343177 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.343133 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:15.343118432 +0000 UTC m=+4.141443832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:14.444423 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.443736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:14.444423 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.443987 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:14.444423 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.444013 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:14.444423 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.444029 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:14.444423 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:14.444088 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:15.444070014 +0000 UTC m=+4.242395438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:14.672687 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.672619 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-07-24 15:25:04.54901243 +0000 UTC" Oct 30 03:02:14.672687 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.672688 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15204h22m49.876331269s" Oct 30 03:02:14.755229 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.754857 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" event={"ID":"2eace382-e677-4a34-9e4e-8c1cd06d0eca","Type":"ContainerStarted","Data":"d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940"} Oct 30 03:02:14.757382 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.757286 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nv6kb" event={"ID":"d76844ce-1723-4f25-af95-a7bf084f3d7c","Type":"ContainerStarted","Data":"53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f"} Oct 30 03:02:14.758806 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.758738 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerStarted","Data":"b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4"} Oct 30 03:02:14.760609 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.760530 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r95wv" event={"ID":"706f817b-2517-45e0-8fb5-75cdb54bfd8c","Type":"ContainerStarted","Data":"9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828"} Oct 30 03:02:14.761978 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.761905 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6xf9" event={"ID":"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea","Type":"ContainerStarted","Data":"5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f"} Oct 30 03:02:14.763711 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.763605 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76"} Oct 30 03:02:14.765217 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.765145 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bm2cm" event={"ID":"a0ab58b7-145a-4e99-b95d-31b89c2c29a0","Type":"ContainerStarted","Data":"cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73"} Oct 30 03:02:14.766955 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.766887 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfts7" event={"ID":"a252df28-aacb-4d80-a585-1f812bf77f0c","Type":"ContainerStarted","Data":"45e996f216ba395a4fc976fd54ac8ea8013e1ecc8de931521f741d8b6535eef1"} Oct 30 03:02:14.770035 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.770012 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fc90fa10f5e4f787096a878d9a85228" containerID="aea29413a689ef03831ca8fa83d8045c96621696397487ed325266b023c70325" exitCode=0 Oct 30 03:02:14.770135 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.770097 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerDied","Data":"aea29413a689ef03831ca8fa83d8045c96621696397487ed325266b023c70325"} Oct 30 03:02:14.774418 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.773969 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"8da5b42e37aadbbb4c814ec87d9b92a3d6027108dfcda7220dd8b50787aaded7"} Oct 30 03:02:14.778239 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.778190 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fv2ns" event={"ID":"cdf7caae-56b4-4610-878e-3dd5a3dda185","Type":"ContainerStarted","Data":"40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07"} Oct 30 03:02:14.850945 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:14.850871 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" podStartSLOduration=2.850852538 podStartE2EDuration="2.850852538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:14.850363657 +0000 UTC m=+3.648689076" watchObservedRunningTime="2026-10-30 03:02:14.850852538 +0000 UTC m=+3.649177958" Oct 30 03:02:15.352550 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.352460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:15.352717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.352586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:15.352782 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.352727 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:15.352836 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.352784 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:17.35276612 +0000 UTC m=+6.151091524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:15.353196 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.353178 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:15.353289 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.353231 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:17.353216297 +0000 UTC m=+6.151541729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:15.453081 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.453050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:15.453254 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.453181 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:15.453254 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.453195 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:15.453254 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.453205 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:15.453254 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.453245 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:17.453233078 +0000 UTC m=+6.251558478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:15.673737 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.673619 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-06-08 16:52:12.657609125 +0000 UTC" Oct 30 03:02:15.673737 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.673683 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14101h49m56.983930362s" Oct 30 03:02:15.745559 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.745508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:15.745737 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.745662 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:15.746136 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.746081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:15.746227 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.746171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:15.746286 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.746240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:15.746338 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:15.746307 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:15.798604 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:15.798568 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"278a7f7a72b557df111939da7861c8069ec87d05e3749c10f1477d3e46460a58"} Oct 30 03:02:17.370067 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.370028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:17.370472 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.370105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:17.370472 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.370248 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:17.370472 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.370315 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:21.370296374 +0000 UTC m=+10.168621993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:17.370472 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.370383 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:17.370472 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.370414 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:21.370404014 +0000 UTC m=+10.168729415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:17.471028 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.470985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:17.471201 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.471162 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:17.471201 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.471185 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:17.471201 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.471197 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:17.471362 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.471253 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:21.471235919 +0000 UTC m=+10.269561324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:17.743440 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.743241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:17.743440 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.743289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:17.743440 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.743241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:17.743440 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.743379 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:17.743753 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.743469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:17.743753 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:17.743576 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:17.810462 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.810429 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="b3ae0fb64464c60bc416b41119a45570308be2a4a925a7af29dd3063df4e8fcf" exitCode=0 Oct 30 03:02:17.810629 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.810486 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"b3ae0fb64464c60bc416b41119a45570308be2a4a925a7af29dd3063df4e8fcf"} Oct 30 03:02:17.877333 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:17.877080 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" podStartSLOduration=5.877059115 podStartE2EDuration="5.877059115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:15.837142712 +0000 UTC m=+4.635468134" watchObservedRunningTime="2026-10-30 03:02:17.877059115 +0000 UTC m=+6.675384537" Oct 30 03:02:18.821650 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:18.821609 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r95wv" event={"ID":"706f817b-2517-45e0-8fb5-75cdb54bfd8c","Type":"ContainerStarted","Data":"c78dfbde9317a2a6fb298204942fa37a67ff7aea731470c5d7de3c5196b6733c"} Oct 30 03:02:19.500814 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.500779 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:19.501438 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.501418 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:19.539930 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.539868 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r95wv" podStartSLOduration=3.731822528 podStartE2EDuration="7.539847044s" totalImagesPullingTime="3.808024516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:18.88868428 +0000 UTC m=+7.687009701" watchObservedRunningTime="2026-10-30 03:02:19.539847044 +0000 UTC m=+8.338172467" Oct 30 03:02:19.744337 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.744305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:19.744503 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:19.744442 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:19.744809 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.744644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:19.744809 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:19.744761 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:19.744993 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.744933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:19.745077 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:19.745057 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:19.830379 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:19.830345 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6xf9" event={"ID":"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea","Type":"ContainerStarted","Data":"fdb5030117c391191cbe3c35db9270c5dbcc13e65b214be9c08b121ddd1f9e52"} Oct 30 03:02:20.837500 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:20.837465 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" event={"ID":"2eace382-e677-4a34-9e4e-8c1cd06d0eca","Type":"ContainerStarted","Data":"326eb3672b833521826d12d2321b1c30d92e980f3871088ab207006e76ddeff5"} Oct 30 03:02:20.837949 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:20.837583 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:21.306057 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.306035 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:02:21.404742 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.404658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:21.404742 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.404728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:21.404966 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.404877 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:21.404966 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.404942 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:29.404922887 +0000 UTC m=+18.203248292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:21.405312 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.405293 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:21.405364 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.405349 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:29.405334109 +0000 UTC m=+18.203659526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:21.505936 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.505898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:21.506117 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.506058 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:21.506117 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.506080 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:21.506117 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.506092 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:21.506264 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.506148 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:29.506131391 +0000 UTC m=+18.304456811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:21.691368 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.691195 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:21.306056384Z","UUID":"42922f0d-0c40-4a00-80d0-dc0034f59098","Handler":null,"Name":"","Endpoint":""} Oct 30 03:02:21.692756 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.692729 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:02:21.692890 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.692762 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:02:21.744056 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.744030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:21.744196 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.744168 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:21.744554 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.744535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:21.744658 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.744637 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:21.744715 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.744706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:21.744788 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:21.744769 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:21.841231 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.841195 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fv2ns" event={"ID":"cdf7caae-56b4-4610-878e-3dd5a3dda185","Type":"ContainerStarted","Data":"33f470b2498c21bd59e0b06aa0f5827412b93fd926bdd3a7c5382f1a7ff74d0c"} Oct 30 03:02:21.844822 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.844795 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" event={"ID":"2eace382-e677-4a34-9e4e-8c1cd06d0eca","Type":"ContainerStarted","Data":"fe4e78a42d53c0fcfb43f3affe6ce2a33ad4f68275ce5650d78dd0e87712622a"} Oct 30 03:02:21.846695 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.846672 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nv6kb" event={"ID":"d76844ce-1723-4f25-af95-a7bf084f3d7c","Type":"ContainerStarted","Data":"8e07c0222ff9f48dc5d0401b2adf87dc1751c7b818b5cca9144b314dafd32923"} Oct 30 03:02:21.879351 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.879293 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fv2ns" podStartSLOduration=2.286284953 podStartE2EDuration="9.879275728s" totalImagesPullingTime="7.592990775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:21.878691755 +0000 UTC m=+10.677017176" watchObservedRunningTime="2026-10-30 03:02:21.879275728 +0000 UTC m=+10.677601153" Oct 30 03:02:21.879494 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.879397 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q6xf9" podStartSLOduration=5.159591243 podStartE2EDuration="9.879390469s" totalImagesPullingTime="4.719799226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:19.868278733 +0000 UTC m=+8.666604156" watchObservedRunningTime="2026-10-30 03:02:21.879390469 +0000 UTC m=+10.677715892" Oct 30 03:02:21.918241 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:21.918189 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nv6kb" podStartSLOduration=2.899946235 podStartE2EDuration="9.91817158s" totalImagesPullingTime="7.018225345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:21.918068883 +0000 UTC m=+10.716394304" watchObservedRunningTime="2026-10-30 03:02:21.91817158 +0000 UTC m=+10.716497003" Oct 30 03:02:22.851866 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:22.851574 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" event={"ID":"2eace382-e677-4a34-9e4e-8c1cd06d0eca","Type":"ContainerStarted","Data":"b141487ee27ad73ca3de2436b3cf7f9de69bb345ad024a910995a4687dbcd9b0"} Oct 30 03:02:22.853529 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:22.853486 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="25fe6559b8c4473b59bda57d3969162272fe52751f115a79f2545193657f2b00" exitCode=0 Oct 30 03:02:22.853658 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:22.853574 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"25fe6559b8c4473b59bda57d3969162272fe52751f115a79f2545193657f2b00"} Oct 30 03:02:22.892661 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:22.892598 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" podStartSLOduration=3.037010613 podStartE2EDuration="10.892581653s" totalImagesPullingTime="7.85557104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.892321905 +0000 UTC m=+11.690647348" watchObservedRunningTime="2026-10-30 03:02:22.892581653 +0000 UTC m=+11.690907075" Oct 30 03:02:23.742599 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.742571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:23.742599 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.742588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:23.742758 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:23.742720 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:23.742886 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:23.742857 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:23.743051 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.742922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:23.743051 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:23.742998 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:23.857087 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.857054 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bm2cm" event={"ID":"a0ab58b7-145a-4e99-b95d-31b89c2c29a0","Type":"ContainerStarted","Data":"f8cc2b069a7989494eb7d2ccdd0602c2867dafc6f79ea6204e0a9926f477ae3e"} Oct 30 03:02:23.858800 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.858772 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfts7" event={"ID":"a252df28-aacb-4d80-a585-1f812bf77f0c","Type":"ContainerStarted","Data":"c6db2a42a9ce137d6c26a816922704d74c4357fcd3130803b50140021041cf16"} Oct 30 03:02:23.861107 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.861086 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="d3e3dba63bb3b0c11a29734574d8e3b5abd0c7392eaeb37325174a494ae5fea6" exitCode=0 Oct 30 03:02:23.861210 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.861174 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"d3e3dba63bb3b0c11a29734574d8e3b5abd0c7392eaeb37325174a494ae5fea6"} Oct 30 03:02:23.891918 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.891871 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bm2cm" podStartSLOduration=5.138624379 podStartE2EDuration="12.891856905s" totalImagesPullingTime="7.753232526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:23.891671075 +0000 UTC m=+12.689996497" watchObservedRunningTime="2026-10-30 03:02:23.891856905 +0000 UTC m=+12.690182326" Oct 30 03:02:23.929717 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:23.929665 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vfts7" podStartSLOduration=3.151488585 podStartE2EDuration="12.929650357s" totalImagesPullingTime="9.778161772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:23.929305479 +0000 UTC m=+12.727630901" watchObservedRunningTime="2026-10-30 03:02:23.929650357 +0000 UTC m=+12.727975777" Oct 30 03:02:24.866906 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:24.866829 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="f8b32ce4ee83a0c6ccc731f63fc4571fbe3f1d2e72bca0d4ce6af1ef3b2a644a" exitCode=0 Oct 30 03:02:24.867490 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:24.867117 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"f8b32ce4ee83a0c6ccc731f63fc4571fbe3f1d2e72bca0d4ce6af1ef3b2a644a"} Oct 30 03:02:25.742411 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:25.742373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:25.742637 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:25.742373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:25.742637 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:25.742496 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:25.742637 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:25.742538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:25.742637 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:25.742629 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:25.742823 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:25.742691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:26.874634 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:26.874600 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="3f2417a511e9f7d9202bbf2b8d4a8c81638564ec3746e144aadaf1532894c23f" exitCode=0 Oct 30 03:02:26.875169 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:26.874652 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"3f2417a511e9f7d9202bbf2b8d4a8c81638564ec3746e144aadaf1532894c23f"} Oct 30 03:02:27.742858 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:27.742643 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:27.743043 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:27.742643 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:27.743043 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:27.742976 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:27.743150 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:27.743041 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:27.743150 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:27.742650 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:27.743234 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:27.743155 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:27.879227 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:27.879192 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="1b56be3a9bfbe56899e4a72a21eabb2af950292d321885d4586924495782ca85" exitCode=0 Oct 30 03:02:27.879630 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:27.879235 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerDied","Data":"1b56be3a9bfbe56899e4a72a21eabb2af950292d321885d4586924495782ca85"} Oct 30 03:02:28.884829 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.884730 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" event={"ID":"bb7e1221-1e48-4518-8b72-f0d6876b9e95","Type":"ContainerStarted","Data":"5e939bf2f8d87eb8a962414e069b9384167de1d37b15f32848bbaeeafbb78f43"} Oct 30 03:02:28.887480 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887456 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"83af6f19cebcb786657565e69813398e43e068e0b1a063c056191fc140b81bf7"} Oct 30 03:02:28.887588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887528 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"285eb97f8b4bd1216103214ae0ff005022f8352dfe2e34a2e7bb18013f7deaf6"} Oct 30 03:02:28.887588 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887563 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"19801dc1fc8708d54e646c9a798026c6e37010e4a8908d23e7a69f9ef734aca6"} Oct 30 03:02:28.887664 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887592 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"8c95a5857ada3d32e8c65396fc4ae4c9ec24dd161de4bf43ad3b2d18bf5d4c0d"} Oct 30 03:02:28.887664 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887620 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"1fa5b04cda1c1cded9adb0d86de8b4ab27dbc75b94c9ec4f2b3d602f713fa48b"} Oct 30 03:02:28.887664 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.887647 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"86dc920caac2bb34da57162389372fce2e401986f591a7b84392cab11848fa6d"} Oct 30 03:02:28.950878 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:28.950823 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kqbn2" podStartSLOduration=7.250535941 podStartE2EDuration="16.950806426s" totalImagesPullingTime="9.700270485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:28.950281427 +0000 UTC m=+17.748606848" watchObservedRunningTime="2026-10-30 03:02:28.950806426 +0000 UTC m=+17.749131848" Oct 30 03:02:29.463111 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.463075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:29.463304 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.463123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:29.463304 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.463220 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:29.463304 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.463231 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:29.463304 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.463270 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret podName:f98b781c-c557-4ac9-a65b-96cf4902994a nodeName:}" failed. No retries permitted until 2026-10-30 03:02:45.463257568 +0000 UTC m=+34.261582980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret") pod "global-pull-secret-syncer-sgjx4" (UID: "f98b781c-c557-4ac9-a65b-96cf4902994a") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:29.463304 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.463293 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:45.463277494 +0000 UTC m=+34.261602908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:29.564210 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.564168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:29.564349 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.564301 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:29.564349 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.564314 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:29.564349 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.564323 2581 projected.go:196] Error preparing data for projected volume kube-api-access-6nncc for pod openshift-network-diagnostics/network-check-target-jmvnr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:29.564473 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.564370 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc podName:48c16a1f-34d1-4934-9e9a-634b1c6e7c40 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:45.564356574 +0000 UTC m=+34.362681972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nncc" (UniqueName: "kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc") pod "network-check-target-jmvnr" (UID: "48c16a1f-34d1-4934-9e9a-634b1c6e7c40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:29.741901 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.741815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:29.741901 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.741869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:29.741901 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:29.741882 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:29.742141 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.741944 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:29.742141 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.742015 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:29.742141 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:29.742087 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:30.895067 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:30.895032 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"1e3f5459d5510adf6e3ec6c10882ae87549021b220fe703ea9634114e0490357"} Oct 30 03:02:31.743742 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:31.743706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:31.743894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:31.743789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:31.743894 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:31.743819 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:31.743894 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:31.743833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:31.744056 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:31.743901 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:31.744056 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:31.743994 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:32.085581 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:32.085488 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:32.086091 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:32.085623 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:32.086091 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:32.086060 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:32.903456 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:32.903403 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" event={"ID":"fe9b309a-c095-4296-8c2c-18059f0785e2","Type":"ContainerStarted","Data":"38c5430911ae3396a70119403fa8d9f118b4ecd666284f30642debe7fa6905ca"} Oct 30 03:02:32.987631 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:32.987545 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" podStartSLOduration=7.653722209 podStartE2EDuration="21.98751239s" totalImagesPullingTime="14.333790181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:32.986999657 +0000 UTC m=+21.785325089" watchObservedRunningTime="2026-10-30 03:02:32.98751239 +0000 UTC m=+21.785837811" Oct 30 03:02:33.741837 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.741802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:33.742436 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.741849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:33.742436 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.741807 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:33.742436 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:33.741937 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:33.742436 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:33.742003 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:33.742436 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:33.742087 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:33.906006 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.905977 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:33.906368 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.906348 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:33.906447 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.906435 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:33.921305 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.921282 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:33.925938 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:33.925918 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:34.811184 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.810937 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgjx4"] Oct 30 03:02:34.811633 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.811328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:34.811633 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:34.811442 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:34.813563 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.813542 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-755m9"] Oct 30 03:02:34.813670 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.813635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:34.813727 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:34.813712 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:34.814006 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.813980 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmvnr"] Oct 30 03:02:34.814077 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.814066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:34.814155 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:34.814139 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:34.907536 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:34.907493 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:35.133816 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:35.133786 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:36.741596 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:36.741567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:36.742042 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:36.741603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:36.742042 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:36.741676 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:36.742042 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:36.741732 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:36.742042 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:36.741750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:36.742042 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:36.741832 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:38.742356 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:38.742327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:38.742890 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:38.742327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:38.742890 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:38.742461 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:38.742890 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:38.742327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:38.742890 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:38.742529 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:38.742890 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:38.742595 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:40.742061 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:40.742026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:40.742463 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:40.742063 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:40.742463 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:40.742122 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmvnr" podUID="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" Oct 30 03:02:40.742463 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:40.742182 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgjx4" podUID="f98b781c-c557-4ac9-a65b-96cf4902994a" Oct 30 03:02:40.742463 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:40.742267 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:40.742463 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:40.742350 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-755m9" podUID="98de80e8-fc98-4ae3-b38c-444671d1eda5" Oct 30 03:02:41.467410 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.467378 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeReady" Oct 30 03:02:41.467577 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.467537 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 03:02:41.522671 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.522636 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fh4pc"] Oct 30 03:02:41.595549 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.595498 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qsmwl"] Oct 30 03:02:41.595725 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.595565 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.607767 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.607745 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fh4pc"] Oct 30 03:02:41.607931 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.607781 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qsmwl"] Oct 30 03:02:41.607931 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.607854 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.613640 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.613620 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:02:41.613745 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.613655 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:02:41.613837 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.613819 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\"" Oct 30 03:02:41.646828 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.646795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.646981 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.646844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwhj\" (UniqueName: \"kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.646981 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.646870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.646981 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.646961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.713911 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.713823 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:02:41.713911 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.713845 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:02:41.713911 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.713889 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:02:41.714110 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.713925 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\"" Oct 30 03:02:41.747716 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.747683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwhj\" (UniqueName: \"kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.747729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.747768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.747796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwl5r\" (UniqueName: \"kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:41.747879 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:41.747932 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.247917142 +0000 UTC m=+31.046242544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.747948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.748141 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.748013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.748426 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.748265 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.748471 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.748454 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.767867 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.767834 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwhj\" (UniqueName: \"kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:41.848900 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.848793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.849074 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.848925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwl5r\" (UniqueName: \"kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.851281 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.851256 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.868018 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.867993 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwl5r\" (UniqueName: \"kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:41.941630 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:41.941602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:42.251871 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.251801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:42.252005 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:42.251937 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:42.252040 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:42.252013 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:43.251997058 +0000 UTC m=+32.050322456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:42.283133 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.283099 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qsmwl"] Oct 30 03:02:42.289174 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:42.289141 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0dd8a3_e490_486f_a60c_1d418854c9e9.slice/crio-994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a WatchSource:0}: Error finding container 994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a: Status 404 returned error can't find the container with id 994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a Oct 30 03:02:42.742203 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.742120 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:42.742203 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.742140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:42.742380 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.742326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:42.757170 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757137 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:02:42.757170 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757162 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:02:42.757513 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757200 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\"" Oct 30 03:02:42.757513 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757304 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:02:42.757513 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757356 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\"" Oct 30 03:02:42.757513 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757382 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:02:42.757513 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.757389 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\"" Oct 30 03:02:42.926184 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:42.926145 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qsmwl" event={"ID":"fc0dd8a3-e490-486f-a60c-1d418854c9e9","Type":"ContainerStarted","Data":"994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a"} Oct 30 03:02:43.261823 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:43.261783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:43.262014 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:43.261944 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:43.262079 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:43.262021 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:45.262005063 +0000 UTC m=+34.060330476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:45.277854 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.277819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:45.278227 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:45.277937 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:45.278227 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:45.277993 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:49.277977742 +0000 UTC m=+38.076303156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:45.480255 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.480162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:45.480255 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.480250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:45.480464 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:45.480324 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:45.480464 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:45.480393 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:17.48037493 +0000 UTC m=+66.278700347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:02:45.482549 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.482512 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:45.534800 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.534769 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:45.581490 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.581460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:45.583835 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.583816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:45.811543 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.811455 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:45.841563 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.841513 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgjx4"] Oct 30 03:02:45.847129 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:45.847102 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98b781c_c557_4ac9_a65b_96cf4902994a.slice/crio-3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06 WatchSource:0}: Error finding container 3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06: Status 404 returned error can't find the container with id 3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06 Oct 30 03:02:45.935142 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.935105 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qsmwl" event={"ID":"fc0dd8a3-e490-486f-a60c-1d418854c9e9","Type":"ContainerStarted","Data":"208090fab26e96ec88b515d576e54b9cf21dc4bd345a056ec8da501d1b527cbb"} Oct 30 03:02:45.936333 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.936304 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sgjx4" event={"ID":"f98b781c-c557-4ac9-a65b-96cf4902994a","Type":"ContainerStarted","Data":"3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06"} Oct 30 03:02:45.972882 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:45.972836 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qsmwl" podStartSLOduration=2.145905377 podStartE2EDuration="4.972824603s" totalImagesPullingTime="2.826919226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:45.972333746 +0000 UTC m=+34.770659169" watchObservedRunningTime="2026-10-30 03:02:45.972824603 +0000 UTC m=+34.771150023" Oct 30 03:02:46.136907 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.136836 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmvnr"] Oct 30 03:02:46.141429 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:46.141397 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c16a1f_34d1_4934_9e9a_634b1c6e7c40.slice/crio-7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6 WatchSource:0}: Error finding container 7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6: Status 404 returned error can't find the container with id 7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6 Oct 30 03:02:46.887498 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.887465 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7bhsr"] Oct 30 03:02:46.903078 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.903047 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.914603 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.914578 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bhsr"] Oct 30 03:02:46.925489 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.925383 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\"" Oct 30 03:02:46.925489 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.925383 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:02:46.925489 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.925426 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:02:46.925489 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.925428 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:02:46.925489 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.925470 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:02:46.940098 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.940067 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmvnr" event={"ID":"48c16a1f-34d1-4934-9e9a-634b1c6e7c40","Type":"ContainerStarted","Data":"7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6"} Oct 30 03:02:46.968492 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.968458 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:02:46.989406 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.989371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:46.990377 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.990345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlp7\" (UniqueName: \"kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.990500 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.990428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.990500 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.990450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.990500 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.990474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.990500 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.990493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:46.991354 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:46.991289 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:02:47.026413 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.026379 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whjh9\"" Oct 30 03:02:47.026619 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.026490 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 03:02:47.051820 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.051785 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 03:02:47.091417 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.091589 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.091589 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091464 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.091589 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091539 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.091589 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.091741 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.091741 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.091800 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.091847 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlp7\" (UniqueName: \"kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.091912 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.091871 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.091949 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.091930 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.591914936 +0000 UTC m=+36.390240335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.092067 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.092054 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:47.092108 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.092090 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.592081691 +0000 UTC m=+36.390407090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:47.092108 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.092204 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.091775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.113446 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.113422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlp7\" (UniqueName: \"kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.192241 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.192162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.192241 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.192204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.192241 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.192230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.192446 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.192288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.192797 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.192775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.193081 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.193063 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.193133 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.193090 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.213379 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.213344 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.315572 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.315537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:47.595958 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.595868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.595958 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.595919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:47.596248 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.595996 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:47.596248 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.596061 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.596044736 +0000 UTC m=+37.394370136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:47.596248 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.596079 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.596248 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:47.596131 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.59611533 +0000 UTC m=+37.394440732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.640398 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.640354 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:02:47.747548 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.746778 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8"] Oct 30 03:02:47.777078 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.777045 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8"] Oct 30 03:02:47.777246 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.777168 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:47.791851 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.791824 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hfhbp\"" Oct 30 03:02:47.792028 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.791992 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:47.792150 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.792054 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 03:02:47.899048 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.898959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcbz\" (UniqueName: \"kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz\") pod \"migrator-66df5d496b-hvzv8\" (UID: \"82a392d1-3d05-4ada-b7ac-fb0e588ba95c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:47.945238 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.945197 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" event={"ID":"9dcd506e-f83a-47fe-a199-92a377ba3bca","Type":"ContainerStarted","Data":"a2279899889ea0e5707565633b690be252cbf346549757af568cc18fc55c1e7f"} Oct 30 03:02:47.999577 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:47.999535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcbz\" (UniqueName: \"kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz\") pod \"migrator-66df5d496b-hvzv8\" (UID: \"82a392d1-3d05-4ada-b7ac-fb0e588ba95c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:48.019870 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.019836 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcbz\" (UniqueName: \"kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz\") pod \"migrator-66df5d496b-hvzv8\" (UID: \"82a392d1-3d05-4ada-b7ac-fb0e588ba95c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:48.107210 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.107169 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:48.430351 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.430309 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8"] Oct 30 03:02:48.441251 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:48.441055 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a392d1_3d05_4ada_b7ac_fb0e588ba95c.slice/crio-328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2 WatchSource:0}: Error finding container 328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2: Status 404 returned error can't find the container with id 328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2 Oct 30 03:02:48.605486 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.605438 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:48.606210 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.605491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:48.606210 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:48.605633 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:48.606210 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:48.605724 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.605703251 +0000 UTC m=+39.404028677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:48.606210 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:48.605750 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:48.606210 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:48.605819 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.605801834 +0000 UTC m=+39.404127247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:48.675755 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.675717 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rcpzf"] Oct 30 03:02:48.707641 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.705714 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rcpzf"] Oct 30 03:02:48.707641 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.705817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.720977 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.720727 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 03:02:48.720977 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.720900 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rllb6\"" Oct 30 03:02:48.720977 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.720948 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 03:02:48.722590 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.722030 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 03:02:48.722590 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.722059 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 03:02:48.808337 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.808293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtrl\" (UniqueName: \"kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.808542 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.808418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.808542 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.808459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.909556 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.909502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.910010 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.909578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.910010 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.909620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtrl\" (UniqueName: \"kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.911006 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.910981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.912792 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.912768 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.935166 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.935135 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtrl\" (UniqueName: \"kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:48.960830 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:48.960754 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" event={"ID":"82a392d1-3d05-4ada-b7ac-fb0e588ba95c","Type":"ContainerStarted","Data":"328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2"} Oct 30 03:02:49.030742 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:49.030699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:49.312593 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:49.312499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:49.312750 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:49.312635 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:49.312750 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:49.312694 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:57.312678175 +0000 UTC m=+46.111003575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:49.355006 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:49.354972 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rcpzf"] Oct 30 03:02:49.360370 ip-10-0-136-97 kubenswrapper[2581]: W1030 03:02:49.360337 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738f0388_3c1e_4467_a82c_0f15fbfb136a.slice/crio-10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b WatchSource:0}: Error finding container 10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b: Status 404 returned error can't find the container with id 10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b Oct 30 03:02:49.966018 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:49.965979 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" event={"ID":"738f0388-3c1e-4467-a82c-0f15fbfb136a","Type":"ContainerStarted","Data":"10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b"} Oct 30 03:02:50.623113 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.623080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:50.623216 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.623122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:50.623216 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:50.623200 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:50.623319 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:50.623263 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:50.623319 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:50.623271 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.623251272 +0000 UTC m=+43.421576679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:50.623435 ip-10-0-136-97 kubenswrapper[2581]: E1030 03:02:50.623338 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.623320475 +0000 UTC m=+43.421645879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:50.970828 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.970744 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmvnr" event={"ID":"48c16a1f-34d1-4934-9e9a-634b1c6e7c40","Type":"ContainerStarted","Data":"2c2a6fee9780919fc4dc010165d7c3b81f417034e24de22d8b5eac2cd6648bd8"} Oct 30 03:02:50.970828 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.970809 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:50.972826 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.972781 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" event={"ID":"9dcd506e-f83a-47fe-a199-92a377ba3bca","Type":"ContainerStarted","Data":"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1"} Oct 30 03:02:50.974706 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:50.974634 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" event={"ID":"82a392d1-3d05-4ada-b7ac-fb0e588ba95c","Type":"ContainerStarted","Data":"7b4a9bf58de0c93f1f634fe546e1b482388cb31d701a6b573c03a0af395de1d6"} Oct 30 03:02:51.008364 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:51.008307 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jmvnr" podStartSLOduration=35.554963556 podStartE2EDuration="40.008290374s" totalImagesPullingTime="4.453326818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:51.007280446 +0000 UTC m=+39.805605893" watchObservedRunningTime="2026-10-30 03:02:51.008290374 +0000 UTC m=+39.806615796" Oct 30 03:02:51.085998 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:51.085937 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" podStartSLOduration=2.133040346 podStartE2EDuration="5.085918509s" totalImagesPullingTime="2.952878163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:51.048101449 +0000 UTC m=+39.846426889" watchObservedRunningTime="2026-10-30 03:02:51.085918509 +0000 UTC m=+39.884243931" Oct 30 03:02:51.086331 ip-10-0-136-97 kubenswrapper[2581]: I1030 03:02:51.086303 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" podStartSLOduration=1.9325045950000002 podStartE2EDuration="4.08629492s" totalImagesPullingTime="2.153790325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:51.085147034 +0000 UTC m=+39.883472460" watchObservedRunningTime="2026-10-30 03:02:51.08629492 +0000 UTC m=+39.884620345" Oct 30 03:02:51.910174 ip-10-0-136-97 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 03:02:51.925036 ip-10-0-136-97 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 03:02:51.925247 ip-10-0-136-97 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 03:02:51.925303 ip-10-0-136-97 systemd[1]: kubelet.service: Consumed 2.438s CPU time. Oct 30 03:02:51.937125 ip-10-0-136-97 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:02:52.053507 ip-10-0-136-97 kubenswrapper[4560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:52.053507 ip-10-0-136-97 kubenswrapper[4560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:02:52.053507 ip-10-0-136-97 kubenswrapper[4560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:52.053507 ip-10-0-136-97 kubenswrapper[4560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:52.056981 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.056855 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.056981 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.056976 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.056981 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.056983 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.057179 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.056991 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.057179 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057054 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.057179 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057088 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057222 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057233 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057238 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057244 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057248 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057252 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057256 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057263 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057270 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057275 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057281 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057285 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057290 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057295 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057300 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057305 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057309 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057313 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.057310 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057318 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057322 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057327 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057331 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057335 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057339 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057344 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057348 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057352 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057356 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057361 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057365 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057369 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057374 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057378 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057383 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057387 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057391 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057396 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057400 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.057934 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057404 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057408 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057413 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057418 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057422 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057427 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057432 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057436 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057440 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057444 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057449 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057454 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057457 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057460 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057463 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057466 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057469 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057471 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057474 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057477 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.058670 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057479 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057482 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057485 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057489 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057495 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057503 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057507 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057512 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057532 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057536 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057541 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057544 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057547 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057550 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057555 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057559 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057563 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057568 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057573 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.059425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057577 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057582 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057587 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057592 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057597 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057601 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057607 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057611 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057616 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057621 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057626 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057630 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057635 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057639 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057643 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057648 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057652 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057657 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057662 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.060191 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057665 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057668 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057670 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.057675 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058701 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058711 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058714 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058717 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058720 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058723 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058725 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058728 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058732 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058734 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058738 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058741 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058744 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058747 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058750 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058752 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.060950 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058755 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058758 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058762 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058767 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058771 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058776 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058780 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058785 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058789 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058793 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058797 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058802 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058807 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058811 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058816 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058821 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058826 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058830 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058835 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058839 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.061723 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058844 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058849 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058853 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058857 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058862 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058866 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058870 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058877 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058882 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058887 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058891 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058896 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058901 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058905 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058910 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058913 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058916 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058919 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058922 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.062457 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058926 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058931 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058934 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058937 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058941 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058945 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058949 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058953 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058956 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058961 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058965 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058968 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058971 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058975 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058977 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058980 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058983 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058986 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058988 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.063257 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058991 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.058996 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059000 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059003 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059006 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059009 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059011 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059014 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059017 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059020 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059023 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059025 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059028 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059032 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059036 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059040 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059045 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059048 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059050 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059053 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.064116 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059056 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059059 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059061 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059066 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059068 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059071 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059074 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059076 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059079 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059081 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059085 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059089 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059907 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059921 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059926 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059929 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059934 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059939 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059944 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059949 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.064968 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059952 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059957 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059965 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059973 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059981 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059987 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059992 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.059997 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060002 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060007 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060013 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060018 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060023 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060025 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060028 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060031 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060034 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060036 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060040 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060044 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.065768 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060049 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060053 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060057 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060061 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060066 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060074 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060078 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060084 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060089 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060093 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060098 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060102 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060105 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060108 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060111 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060114 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060116 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060119 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060122 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060124 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.066784 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060127 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060132 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060137 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060142 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060147 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060151 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060155 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060160 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060164 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060168 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060172 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060176 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060180 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060184 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060188 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060193 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060197 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060204 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060208 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.067749 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060211 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060217 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060221 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060225 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060228 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060232 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060237 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060241 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060246 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060255 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060261 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060266 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060272 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060276 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060280 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060283 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060285 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060288 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060293 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.068597 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060297 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060301 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060306 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060311 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060315 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060318 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060321 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060324 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060327 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060330 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060332 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060337 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060340 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060342 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060345 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060349 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060353 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060359 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060366 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.060370 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.069425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060536 4560 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060551 4560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060563 4560 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060571 4560 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060579 4560 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060582 4560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060587 4560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060594 4560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060599 4560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060605 4560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060611 4560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060617 4560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060623 4560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060628 4560 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060632 4560 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060635 4560 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060638 4560 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060649 4560 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060653 4560 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060656 4560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060660 4560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060663 4560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060667 4560 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060674 4560 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:02:52.070321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060677 4560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060682 4560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060688 4560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060693 4560 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060700 4560 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060706 4560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060711 4560 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060715 4560 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060721 4560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060726 4560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060731 4560 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060735 4560 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060740 4560 flags.go:64] FLAG: --enable-server="true" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060750 4560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060759 4560 flags.go:64] FLAG: --event-burst="100" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060763 4560 flags.go:64] FLAG: --event-qps="50" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060767 4560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060771 4560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060777 4560 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060785 4560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060790 4560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060795 4560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060798 4560 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060801 4560 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060804 4560 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:02:52.071471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060807 4560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060810 4560 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060813 4560 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060817 4560 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060820 4560 flags.go:64] FLAG: --feature-gates="" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060826 4560 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060837 4560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060843 4560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060849 4560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060854 4560 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060857 4560 flags.go:64] FLAG: --help="false" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060861 4560 flags.go:64] FLAG: --hostname-override="ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060865 4560 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060868 4560 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060871 4560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060874 4560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060880 4560 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060885 4560 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060890 4560 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060894 4560 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060903 4560 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060908 4560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060914 4560 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060918 4560 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:02:52.072694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060921 4560 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060925 4560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060930 4560 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060935 4560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060940 4560 flags.go:64] FLAG: --lock-file="" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060945 4560 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060951 4560 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060957 4560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060965 4560 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060969 4560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060972 4560 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060975 4560 flags.go:64] FLAG: --logging-format="text" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060978 4560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060983 4560 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060990 4560 flags.go:64] FLAG: --manifest-url="" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.060994 4560 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061002 4560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061008 4560 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061014 4560 flags.go:64] FLAG: --max-pods="110" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061019 4560 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061025 4560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061031 4560 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061036 4560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061042 4560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061048 4560 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:02:52.073760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061053 4560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061066 4560 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061070 4560 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061075 4560 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061078 4560 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061082 4560 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061086 4560 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061091 4560 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061096 4560 flags.go:64] FLAG: --port="10250" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061102 4560 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061107 4560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00bd38562a7dd02fe" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061111 4560 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061116 4560 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061121 4560 flags.go:64] FLAG: --register-node="true" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061126 4560 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061134 4560 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061139 4560 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061143 4560 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061148 4560 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061154 4560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061159 4560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061165 4560 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061169 4560 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061174 4560 flags.go:64] FLAG: --runonce="false" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061180 4560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061186 4560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061191 4560 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:02:52.074808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061196 4560 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061201 4560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061208 4560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061214 4560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061219 4560 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061224 4560 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061229 4560 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061233 4560 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061240 4560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061246 4560 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061251 4560 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061256 4560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061265 4560 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061270 4560 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061275 4560 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061282 4560 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061286 4560 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061292 4560 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061298 4560 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061304 4560 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061308 4560 flags.go:64] FLAG: --v="2" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061313 4560 flags.go:64] FLAG: --version="false" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061317 4560 flags.go:64] FLAG: --vmodule="" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061322 4560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.061326 4560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:02:52.075672 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061526 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061533 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061537 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061542 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061545 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061547 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061550 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061553 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061556 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061559 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061562 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061565 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061568 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061571 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061575 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061582 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061587 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061590 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061595 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061599 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.076648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061603 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061607 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061611 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061617 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061622 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061627 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061632 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061637 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061641 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061645 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061649 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061653 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061658 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061662 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061666 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061673 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061679 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061682 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061685 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061689 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.077371 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061693 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061698 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061702 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061706 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061710 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061714 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061718 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061724 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061729 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061733 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061736 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061739 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061743 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061747 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061751 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061755 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061762 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061768 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061773 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.078127 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061777 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061784 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061787 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061790 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061793 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061796 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061800 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061804 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061808 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061812 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061816 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061821 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061825 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061830 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061834 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061839 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061842 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061845 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061848 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061853 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.078827 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061858 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061862 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061866 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061870 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061874 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061878 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061883 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061887 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061891 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061895 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061898 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061901 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061904 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061910 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061914 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061918 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061922 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061926 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061930 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061934 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.079594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061938 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061942 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061946 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061951 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061954 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061956 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.061960 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.062002 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062047 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062052 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062056 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062059 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062062 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062066 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062069 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062072 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.080318 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062074 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062077 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062079 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062082 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062085 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062087 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062090 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062093 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062097 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062105 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062109 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062112 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062115 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062118 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062121 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062123 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062126 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062128 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062131 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062135 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.080911 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062139 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062144 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062148 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062152 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062157 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062161 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062165 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062178 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062183 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062189 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062194 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062199 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062203 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062207 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062211 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062216 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062221 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062225 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062229 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.081923 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062234 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062239 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062247 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062252 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062256 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062261 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062266 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062270 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062274 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062277 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062280 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062283 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062286 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062288 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062291 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062294 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062296 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062299 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062302 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062305 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.082393 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062307 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062311 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062318 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062322 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062328 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062334 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062341 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062346 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062350 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062355 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062360 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062364 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062369 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062373 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062382 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062386 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062391 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062395 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062400 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.082933 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062404 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062409 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062413 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062417 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062421 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062425 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062432 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062437 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062442 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062447 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062452 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062456 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062461 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062465 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062469 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062476 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062481 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062485 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062490 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.083556 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.062494 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.067924 4560 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.067942 4560 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.067968 4560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.067976 4560 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068059 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068067 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068072 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068078 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068082 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068087 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068091 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068096 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068100 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068104 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068109 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068114 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068118 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068123 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068128 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.084258 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068132 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068137 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068142 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068146 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068151 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068158 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068164 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068170 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068175 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068181 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068186 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068190 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068195 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068199 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068204 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068209 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068213 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068220 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068225 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.085159 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068231 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068236 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068240 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068245 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068251 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068255 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068260 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068265 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068269 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068273 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068278 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068282 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068287 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068291 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068296 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068301 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068305 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068310 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068314 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068318 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.085874 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068323 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068328 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068334 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068338 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068342 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068347 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068351 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068356 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068360 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068364 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068369 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068373 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068378 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068382 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068386 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068391 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068395 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068400 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068405 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068409 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.086641 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068414 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068418 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068423 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068428 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068432 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068436 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068441 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068446 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068450 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068454 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068459 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068464 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068468 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068473 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068477 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068482 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068486 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068491 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068496 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.087422 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068501 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068505 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068510 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068530 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068535 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068539 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068544 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068549 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068553 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068557 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068561 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068566 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068571 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.068613 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068855 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068863 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.088095 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068867 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068872 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068876 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068881 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068886 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068891 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068896 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068900 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068905 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068909 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068914 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068920 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068926 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068933 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068940 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068945 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068949 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068954 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068959 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.088775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068963 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068968 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068972 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068977 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068981 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068986 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068990 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068995 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.068999 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069004 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069011 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069017 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069023 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069027 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069032 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069036 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069041 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069045 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069050 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.089667 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069055 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069059 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069064 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069068 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069073 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069077 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069081 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069086 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069091 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069096 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069100 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069105 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069109 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069114 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069118 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069122 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069126 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069131 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069135 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069139 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.090398 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069144 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069148 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069153 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069157 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069161 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069165 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069170 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069174 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069178 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069183 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069188 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069192 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069196 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069201 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069205 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069214 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069218 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069223 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069228 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069233 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.091128 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069237 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069243 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069247 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069252 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069256 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069260 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069265 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069269 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069273 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069278 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069282 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069286 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069291 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069295 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069300 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069304 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069308 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069312 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069317 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.091931 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069321 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069326 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069332 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069336 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069340 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069345 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069349 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.069389 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069441 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069450 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069455 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069460 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069464 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069469 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069473 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069485 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:52.092789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069489 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069494 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069499 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069503 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069508 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069529 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069534 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069539 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069543 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069547 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069552 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069556 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069561 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069566 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069570 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069575 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069579 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069583 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069590 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:52.093464 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069596 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069602 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069606 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069611 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069616 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069620 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069627 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069631 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069636 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069641 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069646 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069650 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069655 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069660 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069665 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069670 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069674 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069680 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069687 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:52.094271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069692 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069696 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069702 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069706 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069711 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069715 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069719 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069724 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069728 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069733 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069737 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069741 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069748 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069753 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069757 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069762 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069767 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069771 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069775 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069794 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:52.095102 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069798 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069803 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069807 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069812 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069816 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069820 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069825 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069830 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069835 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069840 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069844 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069848 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069853 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069857 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069861 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069866 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069870 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069874 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069878 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069883 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:52.095941 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069887 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069891 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069896 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069900 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069906 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069912 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069916 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069921 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069925 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069929 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069934 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069942 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069947 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069951 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069955 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069960 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069964 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069969 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069973 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:02:52.069977 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:52.096561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.070062 4560 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.071719 4560 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.071804 4560 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.072073 4560 server.go:1014] "Starting client certificate rotation" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.072226 4560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.072293 4560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:11 +0000 UTC" deadline="2028-05-21 15:26:54.829999914 +0000 UTC" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.072331 4560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13668h24m2.757671408s" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.072834 4560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.074557 4560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.076230 4560 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.078201 4560 log.go:25] "Validated CRI v1 image API" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.079121 4560 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:02:52.097317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.081037 4560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d5b97899-7b6e-4f0a-acb4-9d1e441f77fa:/dev/nvme0n1p3 e7a15d70-85a0-48c1-b892-2e2942631c3e:/dev/nvme0n1p4] Oct 30 03:02:52.097888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.081066 4560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2279899889ea0e5707565633b690be252cbf346549757af568cc18fc55c1e7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2279899889ea0e5707565633b690be252cbf346549757af568cc18fc55c1e7f/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eace382-e677-4a34-9e4e-8c1cd06d0eca/volumes/kubernetes.io~projected/kube-api-access-5w8w5:{mountpoint:/var/lib/kubelet/pods/2eace382-e677-4a34-9e4e-8c1cd06d0eca/volumes/kubernetes.io~projected/kube-api-access-5w8w5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea/volumes/kubernetes.io~projected/kube-api-access-c7mgv:{mountpoint:/var/lib/kubelet/pods/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea/volumes/kubernetes.io~projected/kube-api-access-c7mgv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48c16a1f-34d1-4934-9e9a-634b1c6e7c40/volumes/kubernetes.io~projected/kube-api-access-6nncc:{mountpoint:/var/lib/kubelet/pods/48c16a1f-34d1-4934-9e9a-634b1c6e7c40/volumes/kubernetes.io~projected/kube-api-access-6nncc major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/706f817b-2517-45e0-8fb5-75cdb54bfd8c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/706f817b-2517-45e0-8fb5-75cdb54bfd8c/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~projected/kube-api-access-hbtrl:{mountpoint:/var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~projected/kube-api-access-hbtrl major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82a392d1-3d05-4ada-b7ac-fb0e588ba95c/volumes/kubernetes.io~projected/kube-api-access-clcbz:{mountpoint:/var/lib/kubelet/pods/82a392d1-3d05-4ada-b7ac-fb0e588ba95c/volumes/kubernetes.io~projected/kube-api-access-clcbz major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db/volumes/kubernetes.io~projected/kube-api-access-7mlp7:{mountpoint:/var/lib/kubelet/pods/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db/volumes/kubernetes.io~projected/kube-api-access-7mlp7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98de80e8-fc98-4ae3-b38c-444671d1eda5/volumes/kubernetes.io~projected/kube-api-access-57jkz:{mountpoint:/var/lib/kubelet/pods/98de80e8-fc98-4ae3-b38c-444671d1eda5/volumes/kubernetes.io~projected/kube-api-access-57jkz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dcd506e-f83a-47fe-a199-92a377ba3bca/volumes/kubernetes.io~projected/kube-api-access-88r9f:{mountpoint:/var/lib/kubelet/pods/9dcd506e-f83a-47fe-a199-92a377ba3bca/volumes/kubernetes.io~projected/kube-api-access-88r9f major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0ab58b7-145a-4e99-b95d-31b89c2c29a0/volumes/kubernetes.io~projected/kube-api-access-xdt2f:{mountpoint:/var/lib/kubelet/pods/a0ab58b7-145a-4e99-b95d-31b89c2c29a0/volumes/kubernetes.io~projected/kube-api-access-xdt2f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~projected/kube-api-access-7lr9l:{mountpoint:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~projected/kube-api-access-7lr9l major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7e1221-1e48-4518-8b72-f0d6876b9e95/volumes/kubernetes.io~projected/kube-api-access-7t945:{mountpoint:/var/lib/kubelet/pods/bb7e1221-1e48-4518-8b72-f0d6876b9e95/volumes/kubernetes.io~projected/kube-api-access-7t945 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdf7caae-56b4-4610-878e-3dd5a3dda185/volumes/kubernetes.io~projected/kube-api-access-kgckr:{mountpoint:/var/lib/kubelet/pods/cdf7caae-56b4-4610-878e-3dd5a3dda185/volumes/kubernetes.io~projected/kube-api-access-kgckr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d76844ce-1723-4f25-af95-a7bf084f3d7c/volumes/kubernetes.io~projected/kube-api-access-w444f:{mountpoint:/var/lib/kubelet/pods/d76844ce-1723-4f25-af95-a7bf084f3d7c/volumes/kubernetes.io~projected/kube-api-access-w444f major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f98b781c-c557-4ac9-a65b-96cf4902994a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f98b781c-c557-4ac9-a65b-96cf4902994a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~projected/kube-api-access-zwl5r:{mountpoint:/var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~projected/kube-api-access-zwl5r major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1/volumes/kubernetes.io~projected/kube-api-access-lbwhj:{mountpoint:/var/lib/kubelet/pods/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1/volumes/kubernetes.io~projected/kube-api-access-lbwhj major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~projected/kube-api-access-2679x:{mountpoint:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~projected/kube-api-access-2679x major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6e272e10e934450400c2b60b3b1e95c860112d9d27fb8402167d7be5682f4739/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0b1e36a4c17016595525dfeacea97b36a9498ebeb961a733db4965ef342d63fe/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c2d64811e6a492cd46009cf5416a5c2fa5a675269d363a9c424c18a974a3e13a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3d159de91d07b860c29deca22880f24be0e36cda3b40954328bc4e96b0b138d9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2f5021c111fc73d752dc12f9c73482a1ada40e52610009e9fe0361161fc40295/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4ec8e436cdbac0683beb78b172f2a12dee5e97e8b3081d586a3b25d45ad7e584/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/845298509b2e18e6f6788f8d006758e98680ff1c926f7a3ab31fba1aeef43ef9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/55e89a25208f1aa8c286dd9407b157185f850afc0efb60813b825741fa1ce376/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fa48682e78f95eb28b4afec6cb6117409d9746003f74a551044cf7bc5a004855/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a9e5b4cdc51863a71a117212533259aa87ab51c9f9aab0ba10ea013bdd588b71/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/00ccd7973c847a71e09ad3755078506c5bc178afe3a96bacbdf420f80a3966af/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/984c0370cc55f87ba8dbe213487c3120f2db4d003808759026e9ce82cac6f907/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9fc45f9590eacf7b1650fa34fd696cb4c43ef2d06603093511b7ab22a98d8285/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5ce4d1f42635db6f44bfe7cc618dbb0f5c42fa434746c1f4c2aa52856fcd84aa/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2dff95cc3e71bf89beec40212ff9333eb37618f9bb7ad469ed1ad3f7134e119c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8638885bb13a342bf923c321f5f7fd92224dde2d1045ccaf4383d83a07862c02/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bc3724b61cc052ce240763754df01024ab7b954afd3dbebe94f8c9162b65a082/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2a3c8409f97b26b51a9d9d06072e80f19d2952d5929a8d3d758f07dd82e9da18/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/79750bb87ecdbfcdb6b839ed17cf3fa1e8a6fd3f1a490ca06c17e03138c238ee/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c260cd2e9b13e165f19ca4445c5a6f99592ce5c1a5a10785d52c4d566e24da49/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/06482313b13912dc0a8532698921a84e8bd5cc56ff29602a629fef79a10a1dbe/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/5c3e9fe8dd465ba8f1fa6d295bba4653e147e7831f6a953deeb0ac55a52acde6/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/e55ae5c7b5bcf3c93b34089a2086196d4f61f22bdea59abe424d8ce3cc029339/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/9dc0e8459b61b9e125ad7c007c178b8b4560e7c010362fc847d29711f4a6d581/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/55e2f5d39a982c7a7c4fa098665603c37418b0cbe53687e9742fae1649580fea/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/663de0450d5804b50c48e812d16fe73d77f12fdf1350bfe91f39578f4e66ea28/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/1f63960ea4cdd3ad4318914640e10c807415861c13fc917e152c220b02d48116/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/b14552e349fc6e42477e0e99743ac999f365e67925533d57557d421e242f7099/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/6bc8122c01e0fd5a06fc2e9d9685c162595cf99ae331253dc0d5f61a0d162566/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/4e8d143c67cb1dd9b081cc3c3987d68c8ff3270b6b6583c39835c798205e5304/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/794cb5287e5c6157c68bb67d2fce7cfa0908025d88b483a2cfda41af7abed94e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/15c487c172f65bf5c11187f776c0fcc959b65e5c41e0fc2c3e6f89552b482a61/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/847b343776fb165052e45aa2ead006762c478ca64c0a0c00f041b6b93218f2c9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d085fc3e3b0490a78c770f15555a1861dd915975823f51b244d3b2f49d5e2f4b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/8e0b8649401cf1628d5dc5d273d60e044d1f320029a2b253b6a6cda71f4e1e1e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/45a9849b2c394cb204617060f0a64cb484a75dd27f7b2b94c91db848ff8f1125/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/16bc709bf7db3803da8071dd53532a20f439b5c26ca40c3b4818b74666c4394e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d149de65834bb8663e8be64af1288de7a2e80b5b06a21e48a4c3db83231b0c7f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c2a108e22c6fbdc6182f2b32dda9573ba59e2709818ca93bcd1aa961768bdb6d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/58638ef0cf5c6392a7df2306b542d3798c0dcf01735b95632729e76d2e4749e8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/725c2b4191c106fbd4f6b2f666849f84479016e15850a587b2c31d6cd80cf543/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/ae8ea346bc2f55915aa19f0596f7d51210ec94f6785436fb551e5153c7d53e73/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.089893 4560 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:52.089275551 +0000 UTC m=+0.074395430 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099341 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f2339cecd52a0852cdbaeffe9e09 SystemUUID:ec24f233-9cec-d52a-0852-cdbaeffe9e09 BootID:5263febe-3f7e-4869-839a-9a45d3c75447 Filesystems:[{Device:/run/containers/storage/overlay-containers/cfbe3ec4238fdbf76d1828bd209d59bc5996d661e252f4f0ad4c2431c27fbc73/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2279899889ea0e5707565633b690be252cbf346549757af568cc18fc55c1e7f/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0ab58b7-145a-4e99-b95d-31b89c2c29a0/volumes/kubernetes.io~projected/kube-api-access-xdt2f DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db/volumes/kubernetes.io~projected/kube-api-access-7mlp7 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~projected/kube-api-access-7lr9l DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d60137751dfec70b04d095518700ae71e1aab7e8055e92f5a55f28011fd6828/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ca866f0a91205da02680db48dbe0769575fc8b6ffeca8aa3434d2899ddbcbc6/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a252df28-aacb-4d80-a585-1f812bf77f0c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b13465626e0c2c29d3e693d4a767e15e01e8faf561fe1638ecf60505af1f7db4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40a205b06d0975a804b79dc24473c4a4ab292668a80dfede66e93f988b05aa07/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1/volumes/kubernetes.io~projected/kube-api-access-lbwhj DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd1d481d0834be405ceb73815213e90c9f9ef1568d72013da5f756249a312c76/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10ad38d397f182003c37114626534ab70cbf46dce8bf65611f5de1e5e6a9fe7b/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7e1221-1e48-4518-8b72-f0d6876b9e95/volumes/kubernetes.io~projected/kube-api-access-7t945 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~projected/kube-api-access-zwl5r DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3346bc10184923b23cfe45b8db53d3473025e30c227e3ac48f92dbc9b7291c06/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~projected/kube-api-access-hbtrl DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1185852289070c638dbbfd12a02c9dd94988c48623a98c8106cfdef2ffc8940/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98de80e8-fc98-4ae3-b38c-444671d1eda5/volumes/kubernetes.io~projected/kube-api-access-57jkz DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d76844ce-1723-4f25-af95-a7bf084f3d7c/volumes/kubernetes.io~projected/kube-api-access-w444f DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea/volumes/kubernetes.io~projected/kube-api-access-c7mgv DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48c16a1f-34d1-4934-9e9a-634b1c6e7c40/volumes/kubernetes.io~projected/kube-api-access-6nncc DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdf7caae-56b4-4610-878e-3dd5a3dda185/volumes/kubernetes.io~projected/kube-api-access-kgckr DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc0dd8a3-e490-486f-a60c-1d418854c9e9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/994ca603623ccf73d7a63b0dc813b14371e67f2b1ea1fb41b325c1d2f3266b9a/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2eace382-e677-4a34-9e4e-8c1cd06d0eca/volumes/kubernetes.io~projected/kube-api-access-5w8w5 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/706f817b-2517-45e0-8fb5-75cdb54bfd8c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fe9b309a-c095-4296-8c2c-18059f0785e2/volumes/kubernetes.io~projected/kube-api-access-2679x DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f98b781c-c557-4ac9-a65b-96cf4902994a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/82a392d1-3d05-4ada-b7ac-fb0e588ba95c/volumes/kubernetes.io~projected/kube-api-access-clcbz DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/328760f007d84d7188e36932e9996d546e465b96c8e30c867e4eb39a0b1f1fe2/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/738f0388-3c1e-4467-a82c-0f15fbfb136a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dcd506e-f83a-47fe-a199-92a377ba3bca/volumes/kubernetes.io~projected/kube-api-access-88r9f DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5541cfb14ed6d1338a070edf9677b23ead996490e66c4c822c83dd6337706f1f/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53b1d678e451105fb4a1cf21799734a0d0d719c484287d402d7a2b7aa7ab9a0f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10ad38d397f1820 MacAddress:26:b8:e0:5a:a5:60 Speed:10000 Mtu:8901} {Name:328760f007d84d7 MacAddress:9e:fb:7c:18:cf:f7 Speed:10000 Mtu:8901} {Name:3346bc10184923b MacAddress:7a:da:9c:02:64:e9 Speed:10000 Mtu:8901} {Name:7ca866f0a91205d MacAddress:d6:0b:21:c9:6a:42 Speed:10000 Mtu:8901} {Name:994ca603623ccf7 MacAddress:ea:99:bc:87:a2:fe Speed:10000 Mtu:8901} {Name:a2279899889ea0e MacAddress:92:7c:59:eb:10:cb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f0:66:74:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:14:e1:81:85:61 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f0:66:74:21 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:4c:b7:16:33:22 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:b0:ad:56:10:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090291 4560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090347 4560 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090735 4560 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090767 4560 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090949 4560 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090958 4560 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.090977 4560 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091006 4560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091150 4560 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091430 4560 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091583 4560 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091627 4560 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091647 4560 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091667 4560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091769 4560 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.091791 4560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.092584 4560 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.092597 4560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:52.098081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.093256 4560 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:02:52.105164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105147 4560 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:02:52.105164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105162 4560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:52.105294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105275 4560 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 03:02:52.105482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105465 4560 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:02:52.105639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105495 4560 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:02:52.105639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105617 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:02:52.105639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105630 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:02:52.105639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105635 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:02:52.105639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105641 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105647 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105653 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105663 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105671 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105678 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105684 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105698 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105709 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105738 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:02:52.105776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.105747 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:02:52.109323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.109307 4560 server.go:1263] "Started kubelet" Oct 30 03:02:52.111347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.109435 4560 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:02:52.111347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.109489 4560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:02:52.111347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.109569 4560 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:02:52.111347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.109839 4560 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:02:52.111347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.110754 4560 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:02:52.109903 ip-10-0-136-97 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.113669 4560 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114345 4560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114463 4560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-07-25 14:47:41.326018494 +0000 UTC" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114495 4560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15227h44m49.211526991s" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114681 4560 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114695 4560 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.114784 4560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.115467 4560 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found" Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.117135 4560 factory.go:55] Registering systemd factory Oct 30 03:02:52.117547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.117151 4560 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:02:52.119110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119045 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcd506e-f83a-47fe-a199-92a377ba3bca" volumeName="kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle" seLinuxMountContext="" Oct 30 03:02:52.119110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119099 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76844ce-1723-4f25-af95-a7bf084f3d7c" volumeName="kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119115 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0dd8a3-e490-486f-a60c-1d418854c9e9" volumeName="kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119129 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eace382-e677-4a34-9e4e-8c1cd06d0eca" volumeName="kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119142 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="738f0388-3c1e-4467-a82c-0f15fbfb136a" volumeName="kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119155 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db" volumeName="kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119199 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b309a-c095-4296-8c2c-18059f0785e2" volumeName="kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119212 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b309a-c095-4296-8c2c-18059f0785e2" volumeName="kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119231 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcd506e-f83a-47fe-a199-92a377ba3bca" volumeName="kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119246 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcd506e-f83a-47fe-a199-92a377ba3bca" volumeName="kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119260 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a252df28-aacb-4d80-a585-1f812bf77f0c" volumeName="kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned" seLinuxMountContext="" Oct 30 03:02:52.119272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119273 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a252df28-aacb-4d80-a585-1f812bf77f0c" volumeName="kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119286 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7e1221-1e48-4518-8b72-f0d6876b9e95" volumeName="kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119299 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0dd8a3-e490-486f-a60c-1d418854c9e9" volumeName="kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119317 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdf7caae-56b4-4610-878e-3dd5a3dda185" volumeName="kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119330 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea" volumeName="kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119343 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98de80e8-fc98-4ae3-b38c-444671d1eda5" volumeName="kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119357 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdf7caae-56b4-4610-878e-3dd5a3dda185" volumeName="kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119369 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b309a-c095-4296-8c2c-18059f0785e2" volumeName="kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119382 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea" volumeName="kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119402 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="706f817b-2517-45e0-8fb5-75cdb54bfd8c" volumeName="kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119416 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="738f0388-3c1e-4467-a82c-0f15fbfb136a" volumeName="kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119428 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ab58b7-145a-4e99-b95d-31b89c2c29a0" volumeName="kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119440 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ab58b7-145a-4e99-b95d-31b89c2c29a0" volumeName="kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119447 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1" volumeName="kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119454 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1" volumeName="kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119462 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a392d1-3d05-4ada-b7ac-fb0e588ba95c" volumeName="kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119470 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7e1221-1e48-4518-8b72-f0d6876b9e95" volumeName="kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119481 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76844ce-1723-4f25-af95-a7bf084f3d7c" volumeName="kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119488 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f98b781c-c557-4ac9-a65b-96cf4902994a" volumeName="kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119496 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="738f0388-3c1e-4467-a82c-0f15fbfb136a" volumeName="kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119541 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db" volumeName="kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119553 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcd506e-f83a-47fe-a199-92a377ba3bca" volumeName="kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119563 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a252df28-aacb-4d80-a585-1f812bf77f0c" volumeName="kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119572 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7e1221-1e48-4518-8b72-f0d6876b9e95" volumeName="kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119580 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7e1221-1e48-4518-8b72-f0d6876b9e95" volumeName="kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119588 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48c16a1f-34d1-4934-9e9a-634b1c6e7c40" volumeName="kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119598 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="706f817b-2517-45e0-8fb5-75cdb54bfd8c" volumeName="kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119612 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76844ce-1723-4f25-af95-a7bf084f3d7c" volumeName="kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119633 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1" volumeName="kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119642 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b309a-c095-4296-8c2c-18059f0785e2" volumeName="kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119650 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b309a-c095-4296-8c2c-18059f0785e2" volumeName="kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x" seLinuxMountContext="" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119662 4560 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:02:52.119758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.119667 4560 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120054 4560 factory.go:153] Registering CRI-O factory Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120068 4560 factory.go:223] Registration of the crio container factory successfully Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120116 4560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120136 4560 factory.go:103] Registering Raw factory Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120151 4560 manager.go:1199] Started watching for new ooms in manager Oct 30 03:02:52.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.120495 4560 manager.go:319] Starting recovery of all containers Oct 30 03:02:52.142640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.142598 4560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:02:52.144328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.144305 4560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:02:52.144460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.144353 4560 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:02:52.144460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.144414 4560 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:02:52.144626 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.144459 4560 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 03:02:52.151359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.151324 4560 generic.go:356] "Generic (PLEG): container finished" podID="3fc90fa10f5e4f787096a878d9a85228" containerID="aea29413a689ef03831ca8fa83d8045c96621696397487ed325266b023c70325" exitCode=0 Oct 30 03:02:52.151740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.151717 4560 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:52.152413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.151918 4560 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:52.152413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.151923 4560 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 03:02:52.165542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.165441 4560 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:52.165947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.165927 4560 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:52.186685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186652 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="1b56be3a9bfbe56899e4a72a21eabb2af950292d321885d4586924495782ca85" exitCode=0 Oct 30 03:02:52.186685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186684 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="3f2417a511e9f7d9202bbf2b8d4a8c81638564ec3746e144aadaf1532894c23f" exitCode=0 Oct 30 03:02:52.186835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186695 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="f8b32ce4ee83a0c6ccc731f63fc4571fbe3f1d2e72bca0d4ce6af1ef3b2a644a" exitCode=0 Oct 30 03:02:52.186835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186705 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="d3e3dba63bb3b0c11a29734574d8e3b5abd0c7392eaeb37325174a494ae5fea6" exitCode=0 Oct 30 03:02:52.186835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186718 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="25fe6559b8c4473b59bda57d3969162272fe52751f115a79f2545193657f2b00" exitCode=0 Oct 30 03:02:52.186835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.186726 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb7e1221-1e48-4518-8b72-f0d6876b9e95" containerID="b3ae0fb64464c60bc416b41119a45570308be2a4a925a7af29dd3063df4e8fcf" exitCode=0 Oct 30 03:02:52.191811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.191786 4560 manager.go:324] Recovery completed Oct 30 03:02:52.207728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207709 4560 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:02:52.207728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207723 4560 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:02:52.207886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207741 4560 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:02:52.207886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207868 4560 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 03:02:52.207941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207880 4560 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 03:02:52.207941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207896 4560 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 03:02:52.207941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207901 4560 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 03:02:52.207941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207907 4560 policy_none.go:50] "Start" Oct 30 03:02:52.208052 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207958 4560 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:02:52.208052 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.207970 4560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:02:52.208113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.208103 4560 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 03:02:52.208142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.208114 4560 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 03:02:52.208142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.208119 4560 policy_none.go:44] "Start" Oct 30 03:02:52.212734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.212102 4560 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:02:52.217785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.217768 4560 manager.go:341] "Starting Device Plugin manager" Oct 30 03:02:52.217898 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.217800 4560 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:02:52.217898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.217811 4560 server.go:87] "Starting device plugin registration server" Oct 30 03:02:52.218114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218100 4560 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:02:52.218172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218115 4560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:02:52.218229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218192 4560 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:02:52.218369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218313 4560 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:02:52.218369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218368 4560 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:02:52.218534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218377 4560 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:02:52.218534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.218440 4560 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:52.218352705Z","UUID":"07ed1861-6c51-4977-8906-17389591c4fb","Handler":null,"Name":"","Endpoint":""} Oct 30 03:02:52.219764 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.219745 4560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:02:52.219871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.219773 4560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:02:52.219871 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.219773 4560 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:02:52.244597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.244554 4560 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"] Oct 30 03:02:52.245025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.244996 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"278a7f7a72b557df111939da7861c8069ec87d05e3749c10f1477d3e46460a58"} Oct 30 03:02:52.245067 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245033 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerDied","Data":"aea29413a689ef03831ca8fa83d8045c96621696397487ed325266b023c70325"} Oct 30 03:02:52.245067 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245049 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"3a0c34a505e88bb7101578db6b0e26d1754acd80d6ffab57c8afe9edd8197240"} Oct 30 03:02:52.245134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245090 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"8da5b42e37aadbbb4c814ec87d9b92a3d6027108dfcda7220dd8b50787aaded7"} Oct 30 03:02:52.245134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245102 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"9800072391f6044275b5b0cbd23db0915938a1640e16edcf3d6b0ef9988780c2"} Oct 30 03:02:52.245227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245212 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.245260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.245225 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.271902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.271873 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:52.272023 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.271923 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.272335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.272322 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:52.272384 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:52.272345 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.318704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.318675 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:52.320034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.320013 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:52.320138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.320048 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:52.320138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.320060 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:52.320138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.320094 4560 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.353503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.353472 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.353635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.353508 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.353635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.353546 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.357895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.357875 4560 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.358008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.357952 4560 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.453934 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.453901 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.454068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.453916 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.454068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.453983 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.454068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.454002 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.454068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.454057 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:52.454209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:52.454072 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:53.093341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.093258 4560 apiserver.go:52] "Watching apiserver" Oct 30 03:02:53.147141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.147104 4560 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:02:53.147626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.147603 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kqbn2","openshift-network-operator/iptables-alerter-bm2cm","openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng","openshift-image-registry/node-ca-q6xf9","openshift-insights/insights-runtime-extractor-7bhsr","openshift-ovn-kubernetes/ovnkube-node-wc8bj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","openshift-ingress-canary/ingress-canary-qsmwl","openshift-multus/network-metrics-daemon-755m9","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal","openshift-dns/dns-default-fh4pc","openshift-multus/multus-nv6kb","openshift-network-diagnostics/network-check-target-jmvnr","openshift-cluster-node-tuning-operator/tuned-vfts7","openshift-service-ca/service-ca-5fffc97bdf-rcpzf","openshift-dns/node-resolver-fv2ns","kube-system/global-pull-secret-syncer-sgjx4","openshift-insights/periodic-gathering-sjqrb-ntgv8","kube-system/konnectivity-agent-r95wv"] Oct 30 03:02:53.152989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.152964 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.155080 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.155060 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.155489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.155474 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.164233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164215 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\"" Oct 30 03:02:53.164339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164275 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:02:53.164384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164339 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.164450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164435 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:02:53.164492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164475 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\"" Oct 30 03:02:53.164542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.164507 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:02:53.165756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.165739 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.165814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.165804 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.166043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166028 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:02:53.166189 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166171 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.166240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166218 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.166240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166223 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.166328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166259 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.166328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166264 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:02:53.166328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166271 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.166328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166287 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\"" Oct 30 03:02:53.166328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166298 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.166472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.166347 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\"" Oct 30 03:02:53.191342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.191320 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:53.191921 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.191906 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:53.214358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.214329 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:53.214449 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.214382 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" Oct 30 03:02:53.214818 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.214805 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:53.214852 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.214840 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" Oct 30 03:02:53.216247 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.216231 4560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:02:53.257108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257074 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdt2f\" (UniqueName: \"kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.257407 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257378 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.257566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257433 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2679x\" (UniqueName: \"kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.257566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257480 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.257661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257578 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.257718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257663 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.257718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257693 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.257807 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257729 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.257981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257890 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.257981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.257934 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.258083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258010 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.258083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258045 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-data-volume\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.258083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258050 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.258083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258043 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-iptables-alerter-script\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.258083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258081 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.258222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258176 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.258255 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258230 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.258287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258257 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.258329 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258309 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.258366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258351 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.258409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258393 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.258446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258433 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mgv\" (UniqueName: \"kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.258484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258469 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.258556 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258500 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.258903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.259009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.258951 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.259057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259007 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.259057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259043 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.259136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259067 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.259181 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259144 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.259223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259177 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lr9l\" (UniqueName: \"kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.259223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259207 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.259304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259237 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.259304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259262 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:53.259304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259292 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.259439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259407 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.259487 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259440 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.259487 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259470 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.259601 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259511 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.259601 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259590 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.259694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259617 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.259694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259658 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.259694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259690 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.259813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259720 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.259813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259751 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.259813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259787 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.259938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259818 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.259985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259949 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.260033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.259983 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgckr\" (UniqueName: \"kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.260033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260015 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.260119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260047 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwhj\" (UniqueName: \"kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.260119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260090 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlp7\" (UniqueName: \"kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.260208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260122 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.260208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260153 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcbz\" (UniqueName: \"kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz\") pod \"migrator-66df5d496b-hvzv8\" (UID: \"82a392d1-3d05-4ada-b7ac-fb0e588ba95c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:53.260208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260184 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:53.260337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260214 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.260337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260237 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.260337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260267 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.260337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260297 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.260337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260334 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260363 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260386 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260415 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260446 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260477 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260503 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260549 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.260583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260579 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260607 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260637 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260882 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260915 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w444f\" (UniqueName: \"kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260944 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.260977 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.261022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261007 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.261354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261038 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8w5\" (UniqueName: \"kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.261354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261065 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.261354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261098 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.261354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261143 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-tuned\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.261354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261168 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf7caae-56b4-4610-878e-3dd5a3dda185-tmp-dir\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.261546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261450 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.261662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.261644 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe9b309a-c095-4296-8c2c-18059f0785e2-ovn-node-metrics-cert\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.262204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262181 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-tmp-dir\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.262446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262426 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.262531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262501 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-config\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.262658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-env-overrides\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.262726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262664 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.262726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262711 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:53.262816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262767 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe9b309a-c095-4296-8c2c-18059f0785e2-ovnkube-script-lib\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.262816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262777 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.262898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262847 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.262898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262891 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.262991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262927 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.262991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262927 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a252df28-aacb-4d80-a585-1f812bf77f0c-tmp\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.262991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.262975 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.263134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263009 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.263134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263041 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.263134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263088 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.263134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263116 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.263321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263151 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t945\" (UniqueName: \"kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.263321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263204 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.263321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263232 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.263321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263251 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.263321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263305 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtrl\" (UniqueName: \"kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.263623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263336 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwl5r\" (UniqueName: \"kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:53.263623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263407 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.263623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263536 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.263623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263569 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.263623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263608 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.263854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.263644 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jkz\" (UniqueName: \"kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.264074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264055 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:02:53.264457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264440 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:02:53.264550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264501 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\"" Oct 30 03:02:53.264743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264726 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hfhbp\"" Oct 30 03:02:53.264822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264804 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:02:53.264822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264816 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.264927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264810 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\"" Oct 30 03:02:53.264927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264830 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:02:53.264927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264921 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264929 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\"" Oct 30 03:02:53.265050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.264994 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:02:53.265050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265015 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265018 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265026 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:02:53.265251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265055 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\"" Oct 30 03:02:53.265251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265075 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265210 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265278 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:02:53.265372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265280 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:02:53.265462 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265447 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 03:02:53.265512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265463 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:02:53.265512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265497 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265555 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265557 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\"" Oct 30 03:02:53.265620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265582 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:02:53.265620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265584 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265603 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265648 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265659 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whjh9\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265667 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265683 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265691 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265723 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265722 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265744 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265761 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265769 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265772 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265789 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265807 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265793 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265835 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265840 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\"" Oct 30 03:02:53.265873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265884 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rllb6\"" Oct 30 03:02:53.266353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.265888 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:02:53.266353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.266013 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\"" Oct 30 03:02:53.268151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.268137 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.268279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.268267 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-config-volume\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.268457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.268442 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.271410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.271385 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-daemon-config\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.271590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.271576 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-cabundle\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.271645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.271626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc0dd8a3-e490-486f-a60c-1d418854c9e9-cert\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:53.271681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.271666 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98b781c-c557-4ac9-a65b-96cf4902994a-original-pull-secret\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.271745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.271731 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-serviceca\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.272474 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.272461 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/738f0388-3c1e-4467-a82c-0f15fbfb136a-signing-key\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.272474 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.272469 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/706f817b-2517-45e0-8fb5-75cdb54bfd8c-konnectivity-ca\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:53.272610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.272598 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.273620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.273606 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.273697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.273680 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/706f817b-2517-45e0-8fb5-75cdb54bfd8c-agent-certs\") pod \"konnectivity-agent-r95wv\" (UID: \"706f817b-2517-45e0-8fb5-75cdb54bfd8c\") " pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:02:53.273731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.273681 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d76844ce-1723-4f25-af95-a7bf084f3d7c-cni-binary-copy\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.277833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.277810 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 03:02:53.278168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.278151 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mgv\" (UniqueName: \"kubernetes.io/projected/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-kube-api-access-c7mgv\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.278303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.278285 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdt2f\" (UniqueName: \"kubernetes.io/projected/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-kube-api-access-xdt2f\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.278345 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.278332 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2679x\" (UniqueName: \"kubernetes.io/projected/fe9b309a-c095-4296-8c2c-18059f0785e2-kube-api-access-2679x\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.278478 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.278463 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.278744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.278729 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwhj\" (UniqueName: \"kubernetes.io/projected/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-kube-api-access-lbwhj\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.280412 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.280397 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcbz\" (UniqueName: \"kubernetes.io/projected/82a392d1-3d05-4ada-b7ac-fb0e588ba95c-kube-api-access-clcbz\") pod \"migrator-66df5d496b-hvzv8\" (UID: \"82a392d1-3d05-4ada-b7ac-fb0e588ba95c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hvzv8" Oct 30 03:02:53.280750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.280735 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lr9l\" (UniqueName: \"kubernetes.io/projected/a252df28-aacb-4d80-a585-1f812bf77f0c-kube-api-access-7lr9l\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.281794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.281776 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlp7\" (UniqueName: \"kubernetes.io/projected/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-api-access-7mlp7\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.281972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.281956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w444f\" (UniqueName: \"kubernetes.io/projected/d76844ce-1723-4f25-af95-a7bf084f3d7c-kube-api-access-w444f\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.282566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.282551 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8w5\" (UniqueName: \"kubernetes.io/projected/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kube-api-access-5w8w5\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.283033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.283019 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgckr\" (UniqueName: \"kubernetes.io/projected/cdf7caae-56b4-4610-878e-3dd5a3dda185-kube-api-access-kgckr\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.283288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.283273 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jkz\" (UniqueName: \"kubernetes.io/projected/98de80e8-fc98-4ae3-b38c-444671d1eda5-kube-api-access-57jkz\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.283539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.283510 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"periodic-gathering-sjqrb-ntgv8\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:02:53.285102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.285085 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwl5r\" (UniqueName: \"kubernetes.io/projected/fc0dd8a3-e490-486f-a60c-1d418854c9e9-kube-api-access-zwl5r\") pod \"ingress-canary-qsmwl\" (UID: \"fc0dd8a3-e490-486f-a60c-1d418854c9e9\") " pod="openshift-ingress-canary/ingress-canary-qsmwl" Oct 30 03:02:53.285234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.285215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t945\" (UniqueName: \"kubernetes.io/projected/bb7e1221-1e48-4518-8b72-f0d6876b9e95-kube-api-access-7t945\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.285267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.285217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtrl\" (UniqueName: \"kubernetes.io/projected/738f0388-3c1e-4467-a82c-0f15fbfb136a-kube-api-access-hbtrl\") pod \"service-ca-5fffc97bdf-rcpzf\" (UID: \"738f0388-3c1e-4467-a82c-0f15fbfb136a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" Oct 30 03:02:53.297900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.297878 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nncc\" (UniqueName: \"kubernetes.io/projected/48c16a1f-34d1-4934-9e9a-634b1c6e7c40-kube-api-access-6nncc\") pod \"network-check-target-jmvnr\" (UID: \"48c16a1f-34d1-4934-9e9a-634b1c6e7c40\") " pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:02:53.364631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364553 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.364631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364589 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364612 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364634 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364656 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364676 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-sys-fs\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-kubelet-config\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364729 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-cnibin\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364725 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364758 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-conf\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364770 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysconfig\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364694 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98b781c-c557-4ac9-a65b-96cf4902994a-dbus\") pod \"global-pull-secret-syncer-sgjx4\" (UID: \"f98b781c-c557-4ac9-a65b-96cf4902994a\") " pod="kube-system/global-pull-secret-syncer-sgjx4" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364812 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-host\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364814 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364812 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-modprobe-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364877 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.364888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364898 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364915 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-etc-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364945 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364923 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364951 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-socket-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364947 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-log-socket\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364970 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.364962 4560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365002 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.364970 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365050 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-lib-modules\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365069 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.865048507 +0000 UTC m=+1.850168396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365077 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-kubelet\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365102 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365104 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365148 4560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365150 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365167 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365193 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.865182661 +0000 UTC m=+1.850302541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365206 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365195 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-crio-socket\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365205 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-multus-certs\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365221 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-systemd\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365245 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea-host\") pod \"node-ca-q6xf9\" (UID: \"34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea\") " pod="openshift-image-registry/node-ca-q6xf9" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365254 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-node-log\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365263 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365279 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365296 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365310 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0ab58b7-145a-4e99-b95d-31b89c2c29a0-host-slash\") pod \"iptables-alerter-bm2cm\" (UID: \"a0ab58b7-145a-4e99-b95d-31b89c2c29a0\") " pod="openshift-network-operator/iptables-alerter-bm2cm" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365316 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365344 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-hostroot\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365346 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365371 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-run\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365376 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365373 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-systemd-units\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365403 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365429 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365438 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-netns\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365454 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365469 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-sysctl-d\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365480 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-run-netns\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365404 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-bin\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365480 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365487 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-ovn\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365507 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365541 4560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365551 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365584 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-var-lib-kubelet\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365597 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.865583735 +0000 UTC m=+1.850703601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365621 4560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365648 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.365614 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.365658 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.865649025 +0000 UTC m=+1.850768894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365676 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365685 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-run-k8s-cni-cncf-io\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365692 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365713 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-multus\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365740 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365745 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-system-cni-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365765 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365789 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365816 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365832 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-device-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365838 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365849 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-kubelet\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365859 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-var-lib-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365862 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365882 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-sys\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365887 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a252df28-aacb-4d80-a585-1f812bf77f0c-etc-kubernetes\") pod \"tuned-vfts7\" (UID: \"a252df28-aacb-4d80-a585-1f812bf77f0c\") " pod="openshift-cluster-node-tuning-operator/tuned-vfts7" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365892 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365916 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365929 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-etc-kubernetes\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365944 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-registration-dir\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365961 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-slash\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365972 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.365996 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366021 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366023 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf7caae-56b4-4610-878e-3dd5a3dda185-hosts-file\") pod \"node-resolver-fv2ns\" (UID: \"cdf7caae-56b4-4610-878e-3dd5a3dda185\") " pod="openshift-dns/node-resolver-fv2ns" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366044 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366137 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-host-var-lib-cni-bin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366147 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366159 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-system-cni-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366138 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-socket-dir-parent\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366174 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366182 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2eace382-e677-4a34-9e4e-8c1cd06d0eca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cjbng\" (UID: \"2eace382-e677-4a34-9e4e-8c1cd06d0eca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cjbng" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366202 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366228 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-openvswitch\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366250 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-host-cni-netd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366255 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366279 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366285 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb7e1221-1e48-4518-8b72-f0d6876b9e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kqbn2\" (UID: \"bb7e1221-1e48-4518-8b72-f0d6876b9e95\") " pod="openshift-multus/multus-additional-cni-plugins-kqbn2" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366301 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-multus-conf-dir\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366283 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d76844ce-1723-4f25-af95-a7bf084f3d7c-cnibin\") pod \"multus-nv6kb\" (UID: \"d76844ce-1723-4f25-af95-a7bf084f3d7c\") " pod="openshift-multus/multus-nv6kb" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366311 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.367379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.366353 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe9b309a-c095-4296-8c2c-18059f0785e2-run-systemd\") pod \"ovnkube-node-wc8bj\" (UID: \"fe9b309a-c095-4296-8c2c-18059f0785e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:53.458204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.458185 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:02:53.870470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.870429 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.870470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.870471 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.870536 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.870575 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870584 4560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870615 4560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870647 4560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870685 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.87065401 +0000 UTC m=+2.855773890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870616 4560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870714 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.87070303 +0000 UTC m=+2.855822903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:02:53.870727 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870729 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.870721369 +0000 UTC m=+2.855841235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:53.870992 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:53.870774 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.870753451 +0000 UTC m=+2.855873332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:53.906481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:53.906458 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fv2ns_cdf7caae-56b4-4610-878e-3dd5a3dda185/dns-node-resolver/0.log" Oct 30 03:02:54.195813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.195731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rcpzf" event={"ID":"738f0388-3c1e-4467-a82c-0f15fbfb136a","Type":"ContainerStarted","Data":"264ff8bea18bdf148ecda088f6ee9b7f54699d6f27bf0191c58b7e17f629fc4a"} Oct 30 03:02:54.705844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.705818 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6xf9_34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea/node-ca/0.log" Oct 30 03:02:54.878658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.878616 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.878672 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.878715 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:54.878744 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878809 4560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878828 4560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:54.878848 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878847 4560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:54.879130 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878895 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.878874735 +0000 UTC m=+4.863994618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:54.879130 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878908 4560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:54.879130 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878917 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.878905308 +0000 UTC m=+4.864025175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:54.879130 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878933 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.878924142 +0000 UTC m=+4.864044008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:02:54.879130 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:54.878948 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.878940488 +0000 UTC m=+4.864060359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:55.712167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:55.712141 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:55.905722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:55.905698 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qsmwl_fc0dd8a3-e490-486f-a60c-1d418854c9e9/serve-healthcheck-canary/0.log" Oct 30 03:02:56.127927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.127855 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:02:56.200836 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.200806 4560 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:56.306096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.306069 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hvzv8_82a392d1-3d05-4ada-b7ac-fb0e588ba95c/migrator/0.log" Oct 30 03:02:56.352110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.352070 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:56.573361 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.573331 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:56.695312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.695285 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:56.893857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.893773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:56.893857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.893813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.893905 4560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.893927 4560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.893943 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.893973 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.89395968 +0000 UTC m=+8.879079545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.893991 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.893982904 +0000 UTC m=+8.879102770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : configmap "kube-rbac-proxy" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.894022 4560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.894064 4560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.894025 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.894066 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.894050387 +0000 UTC m=+8.879170269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:02:56.894266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:02:56.894098 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.894092434 +0000 UTC m=+8.879212300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:02:56.916265 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:56.916211 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.020632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:57.020597 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.122053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:57.122022 4560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:59.893903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:02:59.893868 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:03:00.656452 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.656425 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc8bj" Oct 30 03:03:00.924348 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.924261 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:00.924348 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.924303 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.924358 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.924389 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924502 4560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924509 4560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924511 4560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924581 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs podName:98de80e8-fc98-4ae3-b38c-444671d1eda5 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.924561312 +0000 UTC m=+16.909681201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs") pod "network-metrics-daemon-755m9" (UID: "98de80e8-fc98-4ae3-b38c-444671d1eda5") : secret "metrics-daemon-secret" not found Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924600 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls podName:fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.924588046 +0000 UTC m=+16.909707912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls") pod "dns-default-fh4pc" (UID: "fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1") : secret "dns-default-metrics-tls" not found Oct 30 03:03:00.924806 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:00.924613 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls podName:8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.924607744 +0000 UTC m=+16.909727610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bhsr" (UID: "8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db") : secret "insights-runtime-extractor-tls" not found Oct 30 03:03:00.925077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:00.924915 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:08.925005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.924973 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsppl"] Oct 30 03:03:08.927792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.927775 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:08.938362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.938314 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt"] Oct 30 03:03:08.942102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.942082 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:08.960671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.960643 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 03:03:08.960671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.960645 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 03:03:08.972442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.972417 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-pmzdv\"" Oct 30 03:03:08.972564 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.972446 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hk7k6\"" Oct 30 03:03:08.972564 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.972508 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 03:03:08.972890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.972872 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsppl"] Oct 30 03:03:08.981340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.981319 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt"] Oct 30 03:03:08.985113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.985092 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:08.985209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.985126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:03:08.985294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.985275 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:08.985574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.985549 4560 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:03:08.987676 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.987657 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1-metrics-tls\") pod \"dns-default-fh4pc\" (UID: \"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1\") " pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:08.987752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.987689 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98de80e8-fc98-4ae3-b38c-444671d1eda5-metrics-certs\") pod \"network-metrics-daemon-755m9\" (UID: \"98de80e8-fc98-4ae3-b38c-444671d1eda5\") " pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:03:08.987896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:08.987880 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bhsr\" (UID: \"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db\") " pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:09.054269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.054238 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-755m9" Oct 30 03:03:09.055345 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.055318 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:09.056403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.056379 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bhsr" Oct 30 03:03:09.088562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.086683 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aaa487-607b-489a-b81d-133f5f47baad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-575jt\" (UID: \"94aaa487-607b-489a-b81d-133f5f47baad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:09.088562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.086738 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcmg\" (UniqueName: \"kubernetes.io/projected/70d4fafe-092f-46f4-9f21-3d924bd42122-kube-api-access-2fcmg\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.088562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.086890 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70d4fafe-092f-46f4-9f21-3d924bd42122-tmp\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.187334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.187254 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aaa487-607b-489a-b81d-133f5f47baad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-575jt\" (UID: \"94aaa487-607b-489a-b81d-133f5f47baad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:09.187334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.187291 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcmg\" (UniqueName: \"kubernetes.io/projected/70d4fafe-092f-46f4-9f21-3d924bd42122-kube-api-access-2fcmg\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.187334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.187322 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70d4fafe-092f-46f4-9f21-3d924bd42122-tmp\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.187694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.187673 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70d4fafe-092f-46f4-9f21-3d924bd42122-tmp\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.189648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.189623 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aaa487-607b-489a-b81d-133f5f47baad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-575jt\" (UID: \"94aaa487-607b-489a-b81d-133f5f47baad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:09.207832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.207798 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcmg\" (UniqueName: \"kubernetes.io/projected/70d4fafe-092f-46f4-9f21-3d924bd42122-kube-api-access-2fcmg\") pod \"downloads-6dd789c8b7-gsppl\" (UID: \"70d4fafe-092f-46f4-9f21-3d924bd42122\") " pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.241200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.241175 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:09.252147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.252129 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:09.371995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.371973 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fh4pc"] Oct 30 03:03:09.374092 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:09.374063 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7f1b08_8e5d_478f_be3b_27ac7fc3dfb1.slice/crio-9c19a22a3380edfc654b522c9d9dbec02efa430bbe31da460b17f79c9ff5f88f WatchSource:0}: Error finding container 9c19a22a3380edfc654b522c9d9dbec02efa430bbe31da460b17f79c9ff5f88f: Status 404 returned error can't find the container with id 9c19a22a3380edfc654b522c9d9dbec02efa430bbe31da460b17f79c9ff5f88f Oct 30 03:03:09.378170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.378148 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bhsr"] Oct 30 03:03:09.381841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.381814 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-755m9"] Oct 30 03:03:09.382750 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:09.382725 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4f04fa_bd1a_471e_9abc_7bf1cbd6b0db.slice/crio-caa1b12182e8234e52b6ec220f9aa72b1401ab0209f150b80534164124da4280 WatchSource:0}: Error finding container caa1b12182e8234e52b6ec220f9aa72b1401ab0209f150b80534164124da4280: Status 404 returned error can't find the container with id caa1b12182e8234e52b6ec220f9aa72b1401ab0209f150b80534164124da4280 Oct 30 03:03:09.388071 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:09.388027 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98de80e8_fc98_4ae3_b38c_444671d1eda5.slice/crio-ae42cabecb03c9fd6888f18d1d060b7dde24100a80145cf8185b6e5e6fbf76df WatchSource:0}: Error finding container ae42cabecb03c9fd6888f18d1d060b7dde24100a80145cf8185b6e5e6fbf76df: Status 404 returned error can't find the container with id ae42cabecb03c9fd6888f18d1d060b7dde24100a80145cf8185b6e5e6fbf76df Oct 30 03:03:09.546314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.546276 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsppl"] Oct 30 03:03:09.548925 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:09.548882 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d4fafe_092f_46f4_9f21_3d924bd42122.slice/crio-e663810ee52f7022101161f088616f69503ecb5e45d70270bfe7b5f9ee0f4539 WatchSource:0}: Error finding container e663810ee52f7022101161f088616f69503ecb5e45d70270bfe7b5f9ee0f4539: Status 404 returned error can't find the container with id e663810ee52f7022101161f088616f69503ecb5e45d70270bfe7b5f9ee0f4539 Oct 30 03:03:09.564807 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:09.564783 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt"] Oct 30 03:03:09.568060 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:09.568035 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aaa487_607b_489a_b81d_133f5f47baad.slice/crio-06a0385c4fcc4e439fd1067a85a3a01c2e9f15222957b8595f910c81ae6788bc WatchSource:0}: Error finding container 06a0385c4fcc4e439fd1067a85a3a01c2e9f15222957b8595f910c81ae6788bc: Status 404 returned error can't find the container with id 06a0385c4fcc4e439fd1067a85a3a01c2e9f15222957b8595f910c81ae6788bc Oct 30 03:03:10.238589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.238549 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" event={"ID":"94aaa487-607b-489a-b81d-133f5f47baad","Type":"ContainerStarted","Data":"06a0385c4fcc4e439fd1067a85a3a01c2e9f15222957b8595f910c81ae6788bc"} Oct 30 03:03:10.240427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.240346 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fh4pc" event={"ID":"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1","Type":"ContainerStarted","Data":"9c19a22a3380edfc654b522c9d9dbec02efa430bbe31da460b17f79c9ff5f88f"} Oct 30 03:03:10.246130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.246106 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bhsr" event={"ID":"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db","Type":"ContainerStarted","Data":"953be3f8dfb115e68be4d25418836e3cefa4d606ee7660452d229e2a8213a355"} Oct 30 03:03:10.246239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.246157 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bhsr" event={"ID":"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db","Type":"ContainerStarted","Data":"a8c8f6645a019f3efe08b7d3c578a9ca11d494a0d1d231aa852e7c7d8fd5cb64"} Oct 30 03:03:10.246239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.246180 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bhsr" event={"ID":"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db","Type":"ContainerStarted","Data":"caa1b12182e8234e52b6ec220f9aa72b1401ab0209f150b80534164124da4280"} Oct 30 03:03:10.247596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.247568 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gsppl" event={"ID":"70d4fafe-092f-46f4-9f21-3d924bd42122","Type":"ContainerStarted","Data":"e663810ee52f7022101161f088616f69503ecb5e45d70270bfe7b5f9ee0f4539"} Oct 30 03:03:10.249554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:10.249529 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-755m9" event={"ID":"98de80e8-fc98-4ae3-b38c-444671d1eda5","Type":"ContainerStarted","Data":"ae42cabecb03c9fd6888f18d1d060b7dde24100a80145cf8185b6e5e6fbf76df"} Oct 30 03:03:11.257076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.256590 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-755m9" event={"ID":"98de80e8-fc98-4ae3-b38c-444671d1eda5","Type":"ContainerStarted","Data":"b8b5c1c4ad475c6886ae33abc02f834e82f26cb396048dbe9a33058cdb93a9dd"} Oct 30 03:03:11.257076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.256684 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-755m9" event={"ID":"98de80e8-fc98-4ae3-b38c-444671d1eda5","Type":"ContainerStarted","Data":"aef50d7c783d365563b4223c4c48521dc480474b2df384093ada0f34b74e37d7"} Oct 30 03:03:11.259485 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.259454 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" event={"ID":"94aaa487-607b-489a-b81d-133f5f47baad","Type":"ContainerStarted","Data":"5dedc90ba7a7f82646c7a751b891a57390ae29f71641e62727eec93038cf579f"} Oct 30 03:03:11.261512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.259965 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:11.265508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.265483 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" Oct 30 03:03:11.345141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:11.345079 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-575jt" podStartSLOduration=1.948366177 podStartE2EDuration="3.345066026s" totalImagesPullingTime="1.396699849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:11.344541757 +0000 UTC m=+19.329661641" watchObservedRunningTime="2026-10-30 03:03:11.345066026 +0000 UTC m=+19.330185911" Oct 30 03:03:12.265642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:12.265599 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fh4pc" event={"ID":"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1","Type":"ContainerStarted","Data":"3d3d45e348c1afec569e3c1db1a38f2a2c3db8222a644a7594298e9509144be9"} Oct 30 03:03:12.266099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:12.265766 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:12.266099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:12.265809 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fh4pc" event={"ID":"fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1","Type":"ContainerStarted","Data":"9bd4b6e701df6e2aca5cea19a7e59814950aa141bcf5e91f34f7ec864f808d6a"} Oct 30 03:03:12.267859 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:12.267833 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bhsr" event={"ID":"8f4f04fa-bd1a-471e-9abc-7bf1cbd6b0db","Type":"ContainerStarted","Data":"1724eaac6648a93999b0129fd7c4ad27bbaa533d83bdacf06509ae3f37e2ff3a"} Oct 30 03:03:13.902305 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:13.902266 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r95wv" Oct 30 03:03:16.808823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.808777 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:03:16.812496 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.812469 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.843066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843038 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 03:03:16.843214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843093 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 03:03:16.843214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843169 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 03:03:16.843214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843093 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 03:03:16.843383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843229 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 03:03:16.843383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.843336 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w2gls\"" Oct 30 03:03:16.844167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.844147 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:03:16.940813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.940779 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl82v\" (UniqueName: \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.940989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.940817 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.940989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.940856 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.940989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.940917 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.941154 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.941071 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.941154 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.941137 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:16.941323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:16.941293 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042061 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl82v\" (UniqueName: \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042104 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042154 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042258 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.042710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042683 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.043010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.042975 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.043088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.043023 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.043196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.043167 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.045056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.045037 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.045147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.045127 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.088833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.088754 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl82v\" (UniqueName: \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\") pod \"console-88c9b7875-l4m8f\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.123711 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.123677 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:17.434309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.433782 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:03:17.497758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.497723 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4"] Oct 30 03:03:17.501869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.501846 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.528376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.528331 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 03:03:17.528537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.528413 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-slf9f\"" Oct 30 03:03:17.528537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.528465 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 03:03:17.532058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.531981 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:03:17.532058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.532018 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:03:17.532239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.532216 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 03:03:17.557729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.551076 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4"] Oct 30 03:03:17.645805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.645760 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbks\" (UniqueName: \"kubernetes.io/projected/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-kube-api-access-2rbks\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.645970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.645826 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.646159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.645937 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.646939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.646357 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.718353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.718317 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf"] Oct 30 03:03:17.721535 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.721494 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.747760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.747733 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbks\" (UniqueName: \"kubernetes.io/projected/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-kube-api-access-2rbks\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.747903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.747790 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.747903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.747844 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.747903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.747870 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.748978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.748951 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.750467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.750442 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.750640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.750617 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.758339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.758314 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 03:03:17.758484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.758374 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 03:03:17.758568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.758483 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4vc7w\"" Oct 30 03:03:17.759131 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.759106 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf"] Oct 30 03:03:17.767811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.767785 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 03:03:17.783112 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.783081 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbks\" (UniqueName: \"kubernetes.io/projected/f5409a86-c8b7-4b16-8e85-7d4b1b62f149-kube-api-access-2rbks\") pod \"openshift-state-metrics-7bd7489957-zs8c4\" (UID: \"f5409a86-c8b7-4b16-8e85-7d4b1b62f149\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.784427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.784394 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rscmd"] Oct 30 03:03:17.787714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.787694 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.819532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.819487 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.848828 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nth27\" (UniqueName: \"kubernetes.io/projected/452439f5-083c-48b5-a919-0e5b1f696619-kube-api-access-nth27\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.848885 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.848961 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.849003 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.849040 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.850547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.849075 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/452439f5-083c-48b5-a919-0e5b1f696619-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.857675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.857598 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 03:03:17.857984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.857940 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 03:03:17.858078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.858006 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzhh\"" Oct 30 03:03:17.858684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.858639 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 03:03:17.949757 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949723 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-textfile\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.949915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949775 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-root\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.949915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949846 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth27\" (UniqueName: \"kubernetes.io/projected/452439f5-083c-48b5-a919-0e5b1f696619-kube-api-access-nth27\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.949915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949887 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8388f2bc-bc94-4436-a40b-9d83f051fbb7-kube-api-access-dlq47\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.949915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949913 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-metrics-client-ca\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949934 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-wtmp\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949959 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.949984 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950015 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950033 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-sys\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950061 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.950087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950085 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.950355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950103 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.950355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950125 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/452439f5-083c-48b5-a919-0e5b1f696619-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.950355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.950140 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-tls\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:17.951478 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.951451 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.952018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.951994 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452439f5-083c-48b5-a919-0e5b1f696619-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.952680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.952656 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/452439f5-083c-48b5-a919-0e5b1f696619-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.954589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.954565 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.956306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.956262 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/452439f5-083c-48b5-a919-0e5b1f696619-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:17.969971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:17.969915 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth27\" (UniqueName: \"kubernetes.io/projected/452439f5-083c-48b5-a919-0e5b1f696619-kube-api-access-nth27\") pod \"kube-state-metrics-5dc8d7d89b-djkjf\" (UID: \"452439f5-083c-48b5-a919-0e5b1f696619\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:18.038637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.038600 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" Oct 30 03:03:18.051758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051735 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-textfile\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.051885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-root\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.051885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051823 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8388f2bc-bc94-4436-a40b-9d83f051fbb7-kube-api-access-dlq47\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.051885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-metrics-client-ca\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051877 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-wtmp\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051930 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-root\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051941 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-sys\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.051988 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-sys\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052001 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052257 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052041 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052257 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052077 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-tls\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052257 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052108 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-textfile\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052661 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052709 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8388f2bc-bc94-4436-a40b-9d83f051fbb7-metrics-client-ca\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.052891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.052844 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-wtmp\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.055008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.054982 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.058812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.058793 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8388f2bc-bc94-4436-a40b-9d83f051fbb7-node-exporter-tls\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.086723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.086680 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8388f2bc-bc94-4436-a40b-9d83f051fbb7-kube-api-access-dlq47\") pod \"node-exporter-rscmd\" (UID: \"8388f2bc-bc94-4436-a40b-9d83f051fbb7\") " pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.105139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.104799 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rscmd" Oct 30 03:03:18.152241 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:18.152206 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5409a86_c8b7_4b16_8e85_7d4b1b62f149.slice/crio-68110e4ee714cb74c8608f4a7c841797196c59f6852492bde2d391e0448b2d37 WatchSource:0}: Error finding container 68110e4ee714cb74c8608f4a7c841797196c59f6852492bde2d391e0448b2d37: Status 404 returned error can't find the container with id 68110e4ee714cb74c8608f4a7c841797196c59f6852492bde2d391e0448b2d37 Oct 30 03:03:18.152374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.152312 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4"] Oct 30 03:03:18.290752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.290714 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c9b7875-l4m8f" event={"ID":"0fc4970a-26d2-4f7c-aaf9-597f1f308799","Type":"ContainerStarted","Data":"a1b5a2adf3818eca329efcfaac258e41f2fe983dd7b8db5bd8741bc2543d5f51"} Oct 30 03:03:18.292159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.292084 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rscmd" event={"ID":"8388f2bc-bc94-4436-a40b-9d83f051fbb7","Type":"ContainerStarted","Data":"e9f9e4746ab85d4f60d9320cb02da3f009f36a0c907b4d09c6da53380b1316c8"} Oct 30 03:03:18.299596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.299571 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" event={"ID":"f5409a86-c8b7-4b16-8e85-7d4b1b62f149","Type":"ContainerStarted","Data":"0293fbabd0079b9387d0570210b6654be798f7998ce318bfd4a0a74a904b58a1"} Oct 30 03:03:18.299719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.299627 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" event={"ID":"f5409a86-c8b7-4b16-8e85-7d4b1b62f149","Type":"ContainerStarted","Data":"68110e4ee714cb74c8608f4a7c841797196c59f6852492bde2d391e0448b2d37"} Oct 30 03:03:18.358202 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.358130 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf"] Oct 30 03:03:18.361648 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:18.361585 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452439f5_083c_48b5_a919_0e5b1f696619.slice/crio-d79dda76ba88a61387c1d27bee5d25be1f99dc3af0fcdd19a80c0ff049120edc WatchSource:0}: Error finding container d79dda76ba88a61387c1d27bee5d25be1f99dc3af0fcdd19a80c0ff049120edc: Status 404 returned error can't find the container with id d79dda76ba88a61387c1d27bee5d25be1f99dc3af0fcdd19a80c0ff049120edc Oct 30 03:03:18.586055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.585975 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:03:18.590426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.590399 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.611006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.610976 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 03:03:18.611006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611003 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 03:03:18.611186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611168 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 03:03:18.611222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611194 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 03:03:18.611253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611238 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 03:03:18.611282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611268 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 03:03:18.611321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611311 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fpvml\"" Oct 30 03:03:18.611391 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611383 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 03:03:18.611438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611419 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 03:03:18.611538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.611512 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 03:03:18.617745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.617722 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:03:18.757958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.757915 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpl4\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.757977 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758008 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758038 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758060 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758103 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758139 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758161 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758204 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758233 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758259 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758286 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.758396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.758331 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.859473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859383 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.859473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859455 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859481 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859533 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859566 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859593 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859617 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859657 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859686 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpl4\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859731 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859768 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859798 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.860032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.859823 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.861482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.861130 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.861682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.861653 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.863539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.863037 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.863539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.863474 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.865469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.865402 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.865597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.865540 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.866120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.866093 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.867282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.867254 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.867888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.867848 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.868576 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.868529 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.869061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.869021 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.870683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.870643 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.883089 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.883047 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpl4\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\") pod \"alertmanager-main-0\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:18.919470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:18.919259 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:03:19.307933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.307904 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:03:19.310751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.310694 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" event={"ID":"f5409a86-c8b7-4b16-8e85-7d4b1b62f149","Type":"ContainerStarted","Data":"f49c66cca6a5f87509b0bd6c440677f45fb2cf37dc88e8ec92567585095549dc"} Oct 30 03:03:19.312284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.312253 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" event={"ID":"452439f5-083c-48b5-a919-0e5b1f696619","Type":"ContainerStarted","Data":"d79dda76ba88a61387c1d27bee5d25be1f99dc3af0fcdd19a80c0ff049120edc"} Oct 30 03:03:19.316073 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:19.316049 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d6cd01_0918_4fa2_b6a3_066dd0e84984.slice/crio-bd7cb339530f08e4b75397c52fba58ed37bddfa8ba3613c1f6fdfd9407ff1be7 WatchSource:0}: Error finding container bd7cb339530f08e4b75397c52fba58ed37bddfa8ba3613c1f6fdfd9407ff1be7: Status 404 returned error can't find the container with id bd7cb339530f08e4b75397c52fba58ed37bddfa8ba3613c1f6fdfd9407ff1be7 Oct 30 03:03:19.456480 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.456022 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8689bf44b8-ffjhb"] Oct 30 03:03:19.461598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.460438 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.479317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479139 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 03:03:19.479317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479258 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 03:03:19.479507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479355 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3atiptfurrgvh\"" Oct 30 03:03:19.479507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479427 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 03:03:19.479507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479490 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 03:03:19.479661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479562 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 03:03:19.479717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.479684 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gl82w\"" Oct 30 03:03:19.483834 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.483792 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8689bf44b8-ffjhb"] Oct 30 03:03:19.565258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565172 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565230 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-grpc-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565276 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d72c2a-cfd0-4420-b58f-56a7437b1a78-tmp\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54d72c2a-cfd0-4420-b58f-56a7437b1a78-metrics-client-ca\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565406 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565445 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2k5\" (UniqueName: \"kubernetes.io/projected/54d72c2a-cfd0-4420-b58f-56a7437b1a78-kube-api-access-xf2k5\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565509 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.565857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.565842 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666457 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666512 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-grpc-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666560 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d72c2a-cfd0-4420-b58f-56a7437b1a78-tmp\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666597 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666642 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54d72c2a-cfd0-4420-b58f-56a7437b1a78-metrics-client-ca\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666683 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666720 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2k5\" (UniqueName: \"kubernetes.io/projected/54d72c2a-cfd0-4420-b58f-56a7437b1a78-kube-api-access-xf2k5\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666761 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.666972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.666796 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.668579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.668300 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54d72c2a-cfd0-4420-b58f-56a7437b1a78-metrics-client-ca\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.668579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.668508 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d72c2a-cfd0-4420-b58f-56a7437b1a78-tmp\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.671399 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.670796 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-grpc-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.672086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.672056 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.672201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.672180 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-tls\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.672698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.672661 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.673100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.672995 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.673100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.673068 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54d72c2a-cfd0-4420-b58f-56a7437b1a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.689220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.689165 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2k5\" (UniqueName: \"kubernetes.io/projected/54d72c2a-cfd0-4420-b58f-56a7437b1a78-kube-api-access-xf2k5\") pod \"thanos-querier-8689bf44b8-ffjhb\" (UID: \"54d72c2a-cfd0-4420-b58f-56a7437b1a78\") " pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:19.794781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:19.794539 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:20.152734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.152632 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8689bf44b8-ffjhb"] Oct 30 03:03:20.158899 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:20.158871 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d72c2a_cfd0_4420_b58f_56a7437b1a78.slice/crio-2747588f0fdf9ee188b5f87c2a79746dea6b6b4e0ea81cce85f27556cc11d5d4 WatchSource:0}: Error finding container 2747588f0fdf9ee188b5f87c2a79746dea6b6b4e0ea81cce85f27556cc11d5d4: Status 404 returned error can't find the container with id 2747588f0fdf9ee188b5f87c2a79746dea6b6b4e0ea81cce85f27556cc11d5d4 Oct 30 03:03:20.321395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.321358 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"bd7cb339530f08e4b75397c52fba58ed37bddfa8ba3613c1f6fdfd9407ff1be7"} Oct 30 03:03:20.326326 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.326264 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" event={"ID":"452439f5-083c-48b5-a919-0e5b1f696619","Type":"ContainerStarted","Data":"096893d444d03d1db26926da92caa1253c009b93f5a61fd173c6ecebe736a29a"} Oct 30 03:03:20.326785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.326755 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" event={"ID":"452439f5-083c-48b5-a919-0e5b1f696619","Type":"ContainerStarted","Data":"4c00ebcf450e5f4f25a98c5d3107e5d96ab5027f301977fe36713f6326439cd6"} Oct 30 03:03:20.329192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.329167 4560 generic.go:356] "Generic (PLEG): container finished" podID="8388f2bc-bc94-4436-a40b-9d83f051fbb7" containerID="2209992ab938cd49e61990c7cad587db974b31140e63391fb6d87d3455a0ad12" exitCode=0 Oct 30 03:03:20.329286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.329235 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rscmd" event={"ID":"8388f2bc-bc94-4436-a40b-9d83f051fbb7","Type":"ContainerDied","Data":"2209992ab938cd49e61990c7cad587db974b31140e63391fb6d87d3455a0ad12"} Oct 30 03:03:20.335039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.334869 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"2747588f0fdf9ee188b5f87c2a79746dea6b6b4e0ea81cce85f27556cc11d5d4"} Oct 30 03:03:20.342329 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.341682 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" event={"ID":"f5409a86-c8b7-4b16-8e85-7d4b1b62f149","Type":"ContainerStarted","Data":"c1d7e13a2826273c722556e99f7e29d5651a6349dab96da93dca3911435a2f2a"} Oct 30 03:03:20.431244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:20.430754 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zs8c4" podStartSLOduration=2.16878884 podStartE2EDuration="3.430736184s" totalImagesPullingTime="1.261947344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:20.430644512 +0000 UTC m=+28.415764401" watchObservedRunningTime="2026-10-30 03:03:20.430736184 +0000 UTC m=+28.415856074" Oct 30 03:03:21.350426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.350389 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f"} Oct 30 03:03:21.353984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.353922 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" event={"ID":"452439f5-083c-48b5-a919-0e5b1f696619","Type":"ContainerStarted","Data":"b3351444ed187fb2446b370a3ca8012fd54dfaac7ce563376557683f6aafea93"} Oct 30 03:03:21.358165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.358140 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rscmd" event={"ID":"8388f2bc-bc94-4436-a40b-9d83f051fbb7","Type":"ContainerStarted","Data":"13be899b497a277b2d6188f644e09b407e63530e15c2a8edf45580fbdd750fed"} Oct 30 03:03:21.358267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.358186 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rscmd" event={"ID":"8388f2bc-bc94-4436-a40b-9d83f051fbb7","Type":"ContainerStarted","Data":"2a29318eb11f86e70c34231e0a35ed3db625160ace326b98937fca29869b6b85"} Oct 30 03:03:21.505916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.505506 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-djkjf" podStartSLOduration=2.75291447 podStartE2EDuration="4.505488803s" totalImagesPullingTime="1.752574333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:21.460532719 +0000 UTC m=+29.445652607" watchObservedRunningTime="2026-10-30 03:03:21.505488803 +0000 UTC m=+29.490608693" Oct 30 03:03:21.505916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:21.505712 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rscmd" podStartSLOduration=2.716909911 podStartE2EDuration="4.505701244s" totalImagesPullingTime="1.708812565s" totalInitContainerRuntime="79.978768ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:21.504437959 +0000 UTC m=+29.489557852" watchObservedRunningTime="2026-10-30 03:03:21.505701244 +0000 UTC m=+29.490821132" Oct 30 03:03:22.274710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.274672 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fh4pc" Oct 30 03:03:22.365187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.365146 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c9b7875-l4m8f" event={"ID":"0fc4970a-26d2-4f7c-aaf9-597f1f308799","Type":"ContainerStarted","Data":"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f"} Oct 30 03:03:22.368272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.368240 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f" exitCode=0 Oct 30 03:03:22.368407 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.368322 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f"} Oct 30 03:03:22.474927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.474847 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-88c9b7875-l4m8f" podStartSLOduration=2.090758324 podStartE2EDuration="6.474828543s" totalImagesPullingTime="4.384070219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:22.415322762 +0000 UTC m=+30.400442650" watchObservedRunningTime="2026-10-30 03:03:22.474828543 +0000 UTC m=+30.459948434" Oct 30 03:03:22.875132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.875096 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85f65bfbcf-prgds"] Oct 30 03:03:22.900428 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.900395 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.910174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.910148 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85f65bfbcf-prgds"] Oct 30 03:03:22.923762 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.923733 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 03:03:22.924113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.924060 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 03:03:22.924245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.924120 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 03:03:22.924245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.924168 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fs8kv\"" Oct 30 03:03:22.924245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.924190 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 03:03:22.924397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.924248 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 03:03:22.937381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.937359 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 03:03:22.999461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999408 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999459 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-metrics-client-ca\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999618 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999664 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999737 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-federate-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999821 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999783 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999859 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4hs\" (UniqueName: \"kubernetes.io/projected/7ff79c21-06e7-472a-9401-f8cddd431bdd-kube-api-access-5k4hs\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:22.999910 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:22.999886 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-serving-certs-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101241 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101296 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4hs\" (UniqueName: \"kubernetes.io/projected/7ff79c21-06e7-472a-9401-f8cddd431bdd-kube-api-access-5k4hs\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101327 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-serving-certs-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101368 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101400 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-metrics-client-ca\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101435 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101482 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.101599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.101546 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-federate-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.102776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.102744 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-serving-certs-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.102891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.102791 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-metrics-client-ca\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.103132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.103105 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.104300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.104257 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-federate-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.104769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.104742 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-telemeter-client-tls\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.105107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.105087 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.106113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.106090 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ff79c21-06e7-472a-9401-f8cddd431bdd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.123657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.123626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4hs\" (UniqueName: \"kubernetes.io/projected/7ff79c21-06e7-472a-9401-f8cddd431bdd-kube-api-access-5k4hs\") pod \"telemeter-client-85f65bfbcf-prgds\" (UID: \"7ff79c21-06e7-472a-9401-f8cddd431bdd\") " pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.261781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.261751 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" Oct 30 03:03:23.377128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.376958 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"00fa1d70a75e57126888f11106bffdf3ed6d55377443351737359022f6cea896"} Oct 30 03:03:23.377128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.377041 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"6a806c2126d844d18b05b8fc1428e971b4c564c5be1f0a0e7982a4ef87fcd0b6"} Oct 30 03:03:23.656723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:23.656645 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85f65bfbcf-prgds"] Oct 30 03:03:23.659707 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:23.659674 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff79c21_06e7_472a_9401_f8cddd431bdd.slice/crio-9fdf695d24218713f4155882db9fe34da0aeaaf9bf06e2b61579ecf53ddefee5 WatchSource:0}: Error finding container 9fdf695d24218713f4155882db9fe34da0aeaaf9bf06e2b61579ecf53ddefee5: Status 404 returned error can't find the container with id 9fdf695d24218713f4155882db9fe34da0aeaaf9bf06e2b61579ecf53ddefee5 Oct 30 03:03:24.217682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.217642 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:03:24.238224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.238189 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.273654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.273601 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 03:03:24.273654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.273621 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 03:03:24.273904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.273683 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aq15m7t422gg8\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285839 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285862 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285844 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285873 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285911 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285914 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-54ggx\"" Oct 30 03:03:24.285985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.285949 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 03:03:24.286355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.286012 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 03:03:24.286355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.286049 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 03:03:24.287242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.287218 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:03:24.357696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.357661 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 03:03:24.357871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.357807 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 03:03:24.359297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.359271 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 03:03:24.359427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.359318 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 03:03:24.360314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.360249 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 03:03:24.383459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.383415 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" event={"ID":"7ff79c21-06e7-472a-9401-f8cddd431bdd","Type":"ContainerStarted","Data":"9fdf695d24218713f4155882db9fe34da0aeaaf9bf06e2b61579ecf53ddefee5"} Oct 30 03:03:24.386003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.385969 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"c130f3bc6a133b3370ec1b86a4bd414898243fe2d978d0c7bf97bca5aa215893"} Oct 30 03:03:24.413720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413674 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.413901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413732 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.413901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413761 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.413901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413874 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413915 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.413951 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414085 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414121 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414153 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414197 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414295 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414318 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414333 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414357 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414436 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414485 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414503 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xw2\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414565 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414593 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.414756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.414645 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515491 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515542 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515574 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515610 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515636 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515666 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515717 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515756 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515787 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515810 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.515846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515844 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515876 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515899 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515940 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.515980 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.516007 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.516029 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xw2\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.516060 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.516084 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.516234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.516119 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.517067 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.517036 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.520380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.517558 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.520380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.518413 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.520380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.519339 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.520380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.519927 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.520986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.520959 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.521630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.521601 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.522125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.522099 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.522224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.522135 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.522350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.522331 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.522640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.522590 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.523669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.523645 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.523770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.523738 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.523944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.523927 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.524845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.524824 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.524943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.524917 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.525264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.525236 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.526283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.526258 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.527945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.527901 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.540457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.540422 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xw2\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\") pod \"prometheus-k8s-0\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:24.672771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:24.672697 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:25.239180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.239156 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:03:25.243330 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:25.242480 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e89ec78_0ea0_4799_b7ac_cfc16c2a4a3c.slice/crio-0ea04e6c4e2e06dba0dccefc230816b4e015496d747a2c27e6b5d950279622c7 WatchSource:0}: Error finding container 0ea04e6c4e2e06dba0dccefc230816b4e015496d747a2c27e6b5d950279622c7: Status 404 returned error can't find the container with id 0ea04e6c4e2e06dba0dccefc230816b4e015496d747a2c27e6b5d950279622c7 Oct 30 03:03:25.378808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.377607 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:25.402551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.401850 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="0609b3d051804e4f8d1e10fee33c5758f54c6395b442f32baa2fecbafcb92da0" exitCode=0 Oct 30 03:03:25.403190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.402948 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee"} Oct 30 03:03:25.403190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403059 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912"} Oct 30 03:03:25.403190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403105 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:25.403190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403148 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2"} Oct 30 03:03:25.403190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403185 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"01c9d502f8cbcbea839c3aa7f7f3c761a5d892217dea409621df67a2c0e2e03b"} Oct 30 03:03:25.403461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403223 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"0485bb571d9e0f7f29dcaaa12990fea870daf84b650307fa21a93d732a085d77"} Oct 30 03:03:25.403461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403263 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" event={"ID":"54d72c2a-cfd0-4420-b58f-56a7437b1a78","Type":"ContainerStarted","Data":"f5b713baaa4a124265f1c422aa226538d1c4b32fcbbfde4927ee9835f0eed8ab"} Oct 30 03:03:25.403461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403304 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"0609b3d051804e4f8d1e10fee33c5758f54c6395b442f32baa2fecbafcb92da0"} Oct 30 03:03:25.403461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403350 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"0ea04e6c4e2e06dba0dccefc230816b4e015496d747a2c27e6b5d950279622c7"} Oct 30 03:03:25.403770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.403462 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.409087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.409065 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:25.427138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427029 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.427138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427106 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.427557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427439 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.427787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427768 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.427870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427806 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.427925 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427912 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.428004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.427983 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 03:03:25.428143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.428128 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.428384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.428365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.507295 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.507237 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" podStartSLOduration=2.137053141 podStartE2EDuration="6.507217472s" totalImagesPullingTime="4.370164331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:25.462997976 +0000 UTC m=+33.448117864" watchObservedRunningTime="2026-10-30 03:03:25.507217472 +0000 UTC m=+33.492337361" Oct 30 03:03:25.529179 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529137 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529199 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529225 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529248 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529274 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529308 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529334 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.529613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.529372 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.530213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.530114 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.530213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.530200 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.530393 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.530369 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.530748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.530725 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.531289 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.531267 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.532451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.532407 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.532661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.532638 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.551204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.551165 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\") pod \"console-5567f984fc-lhs5l\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:25.766390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:25.766358 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:26.104070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.104008 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:26.107337 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:26.107299 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978d60b4_621f_4c30_bb10_ddae7ab03ffa.slice/crio-1a330f42298fba937b31f46a627248b15e085eb40738331ff9fa99bc2fd76757 WatchSource:0}: Error finding container 1a330f42298fba937b31f46a627248b15e085eb40738331ff9fa99bc2fd76757: Status 404 returned error can't find the container with id 1a330f42298fba937b31f46a627248b15e085eb40738331ff9fa99bc2fd76757 Oct 30 03:03:26.412425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.411727 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a"} Oct 30 03:03:26.412425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.411803 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106"} Oct 30 03:03:26.412425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.411844 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerStarted","Data":"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15"} Oct 30 03:03:26.414147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.414070 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5567f984fc-lhs5l" event={"ID":"978d60b4-621f-4c30-bb10-ddae7ab03ffa","Type":"ContainerStarted","Data":"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4"} Oct 30 03:03:26.414147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.414117 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5567f984fc-lhs5l" event={"ID":"978d60b4-621f-4c30-bb10-ddae7ab03ffa","Type":"ContainerStarted","Data":"1a330f42298fba937b31f46a627248b15e085eb40738331ff9fa99bc2fd76757"} Oct 30 03:03:26.474746 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.474687 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.017009867 podStartE2EDuration="8.474668891s" totalImagesPullingTime="4.446532617s" totalInitContainerRuntime="11.126407ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:26.473797095 +0000 UTC m=+34.458916984" watchObservedRunningTime="2026-10-30 03:03:26.474668891 +0000 UTC m=+34.459788779" Oct 30 03:03:26.520041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:26.519991 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5567f984fc-lhs5l" podStartSLOduration=1.519978607 podStartE2EDuration="1.519978607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:26.518368518 +0000 UTC m=+34.503488397" watchObservedRunningTime="2026-10-30 03:03:26.519978607 +0000 UTC m=+34.505098495" Oct 30 03:03:27.124307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.124266 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:27.125082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.125058 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:27.130431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.130408 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:27.424252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.423587 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" event={"ID":"7ff79c21-06e7-472a-9401-f8cddd431bdd","Type":"ContainerStarted","Data":"845f4d70a3f71f083b5fc106a21dfcf269b708e3229ee4cbbfb4a73c1ea392ce"} Oct 30 03:03:27.424252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.423652 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" event={"ID":"7ff79c21-06e7-472a-9401-f8cddd431bdd","Type":"ContainerStarted","Data":"5ace41cc3ae6df9af05c90fbe5988d04ea66e836491dd0e6c834820f0f42ead1"} Oct 30 03:03:27.424252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.423683 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" event={"ID":"7ff79c21-06e7-472a-9401-f8cddd431bdd","Type":"ContainerStarted","Data":"70977f4a9f23feed25a49e2107c1154469c8f04ae7bc3332c7f9d19481e2cb5d"} Oct 30 03:03:27.428172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.428144 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:03:27.503984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:27.502479 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85f65bfbcf-prgds" podStartSLOduration=2.790468181 podStartE2EDuration="5.502459432s" totalImagesPullingTime="2.711991251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:27.489635971 +0000 UTC m=+35.474755860" watchObservedRunningTime="2026-10-30 03:03:27.502459432 +0000 UTC m=+35.487579321" Oct 30 03:03:29.436040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.434421 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gsppl" event={"ID":"70d4fafe-092f-46f4-9f21-3d924bd42122","Type":"ContainerStarted","Data":"84c79afd4009d7b1415afd7eb61ab2449671cc998416c239a1bb1a19d5750a9c"} Oct 30 03:03:29.436040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.434883 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:29.436573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.436096 4560 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-gsppl container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.12:8080/\": dial tcp 10.132.0.12:8080: connect: connection refused" start-of-body= Oct 30 03:03:29.436573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.436155 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-gsppl" podUID="70d4fafe-092f-46f4-9f21-3d924bd42122" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.12:8080/\": dial tcp 10.132.0.12:8080: connect: connection refused" Oct 30 03:03:29.445653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.445625 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"c87105013ddb77760f41a91258d96f6b3f7ea9d8fd1d53a732b4e0dcc9578e73"} Oct 30 03:03:29.445787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.445693 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"d9e9cde010559cc0c58ec3fe1d9a54189540908a89ee5f5816cb01763a6e01cd"} Oct 30 03:03:29.495144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.495094 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gsppl" podStartSLOduration=1.801581783 podStartE2EDuration="21.495077923s" totalImagesPullingTime="19.69349614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:29.494216075 +0000 UTC m=+37.479335963" watchObservedRunningTime="2026-10-30 03:03:29.495077923 +0000 UTC m=+37.480197811" Oct 30 03:03:29.897938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:29.897862 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jmvnr" Oct 30 03:03:30.459708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.459653 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"63069d3c76fd76a07c96deef9ebf93b7809bf1ac60319a832d36dc12ad057371"} Oct 30 03:03:30.460180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.459731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"9def3e44c02559f64c2b9c42b7699939a8178170c3c171c3e93b4ef1f3b5f837"} Oct 30 03:03:30.460180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.459769 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"137c2f366f7359de306f5acabbc66f4c614aecbc602144e26fc637a46b67599e"} Oct 30 03:03:30.460180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.459807 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerStarted","Data":"af22b7692efedaf20fa38efee54ee95600fc723b913092ff9ed15c56efdf39a3"} Oct 30 03:03:30.478691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.478657 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gsppl" Oct 30 03:03:30.592852 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:30.592785 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7639868119999997 podStartE2EDuration="6.592765026s" totalImagesPullingTime="3.822220532s" totalInitContainerRuntime="6.557682ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:30.59190584 +0000 UTC m=+38.577025773" watchObservedRunningTime="2026-10-30 03:03:30.592765026 +0000 UTC m=+38.577884915" Oct 30 03:03:31.154045 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.154010 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:31.182490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.182449 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:03:31.210655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.210624 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:03:31.210857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.210754 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292270 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292343 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292456 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292554 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292586 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr96\" (UniqueName: \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292668 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.292767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.292690 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.393822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.393780 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.393848 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.393886 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.393911 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr96\" (UniqueName: \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.393973 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.394009 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.394038 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.394278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.394134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.395125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.394706 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.395125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.394754 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.395125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.395055 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.395125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.395082 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.395541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.395509 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.396987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.396959 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.397849 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.397827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.418640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.418543 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr96\" (UniqueName: \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\") pod \"console-7c4794fd9-kk5nx\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.422457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.422432 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8689bf44b8-ffjhb" Oct 30 03:03:31.546242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.546203 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:31.879634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:31.879597 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:03:31.882446 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:03:31.882415 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164397bb_bbb8_4eab_8391_d31aab597329.slice/crio-00c00a56327fa9d3b1a1e49fc32c790cfdd8dc871f6ca289bcdd3ab70159e43e WatchSource:0}: Error finding container 00c00a56327fa9d3b1a1e49fc32c790cfdd8dc871f6ca289bcdd3ab70159e43e: Status 404 returned error can't find the container with id 00c00a56327fa9d3b1a1e49fc32c790cfdd8dc871f6ca289bcdd3ab70159e43e Oct 30 03:03:32.471085 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:32.471048 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4794fd9-kk5nx" event={"ID":"164397bb-bbb8-4eab-8391-d31aab597329","Type":"ContainerStarted","Data":"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04"} Oct 30 03:03:32.471298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:32.471115 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4794fd9-kk5nx" event={"ID":"164397bb-bbb8-4eab-8391-d31aab597329","Type":"ContainerStarted","Data":"00c00a56327fa9d3b1a1e49fc32c790cfdd8dc871f6ca289bcdd3ab70159e43e"} Oct 30 03:03:32.529983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:32.529927 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c4794fd9-kk5nx" podStartSLOduration=1.529912576 podStartE2EDuration="1.529912576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:32.516337624 +0000 UTC m=+40.501457512" watchObservedRunningTime="2026-10-30 03:03:32.529912576 +0000 UTC m=+40.515032463" Oct 30 03:03:34.675026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:34.673802 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:03:35.767330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:35.767284 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:41.546778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:41.546692 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:41.547300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:41.546815 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:41.551539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:41.551485 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:41.748652 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:41.748615 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:03:42.519919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:42.519888 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:03:42.596457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:42.596423 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:03:56.177483 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.177421 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5567f984fc-lhs5l" podUID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" containerName="console" containerID="cri-o://a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4" gracePeriod=15 Oct 30 03:03:56.416076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.416044 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:56.533730 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533694 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\" (UniqueName: \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.533903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533801 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.533903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533828 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.533903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533846 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.533903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533876 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.534147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533949 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.534147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.533976 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.534147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534000 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:56.534147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534028 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\") pod \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\" (UID: \"978d60b4-621f-4c30-bb10-ddae7ab03ffa\") " Oct 30 03:03:56.534357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534297 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978d60b4-621f-4c30-bb10-ddae7ab03ffa-tmp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.534357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534319 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:56.534454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534383 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:56.534454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534389 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:56.534454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.534425 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:56.536292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.536258 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "kube-api-access-b5k9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:56.536292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.536281 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:03:56.536436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.536355 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config" pod "978d60b4-621f-4c30-bb10-ddae7ab03ffa" (UID: "978d60b4-621f-4c30-bb10-ddae7ab03ffa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:03:56.579615 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.579582 4560 generic.go:356] "Generic (PLEG): container finished" podID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" containerID="a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4" exitCode=0 Oct 30 03:03:56.579775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.579628 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5567f984fc-lhs5l" Oct 30 03:03:56.579775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.579661 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5567f984fc-lhs5l" event={"ID":"978d60b4-621f-4c30-bb10-ddae7ab03ffa","Type":"ContainerDied","Data":"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4"} Oct 30 03:03:56.579775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.579710 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5567f984fc-lhs5l" event={"ID":"978d60b4-621f-4c30-bb10-ddae7ab03ffa","Type":"ContainerDied","Data":"1a330f42298fba937b31f46a627248b15e085eb40738331ff9fa99bc2fd76757"} Oct 30 03:03:56.579775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.579743 4560 scope.go:122] "RemoveContainer" containerID="a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4" Oct 30 03:03:56.588137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.588120 4560 scope.go:122] "RemoveContainer" containerID="a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4" Oct 30 03:03:56.588423 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:03:56.588404 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4\": container with ID starting with a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4 not found: ID does not exist" containerID="a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4" Oct 30 03:03:56.588470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.588435 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4"} err="failed to get container status \"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4\": rpc error: code = NotFound desc = could not find container \"a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4\": container with ID starting with a2e0ff042b5d743c7e0372a5ed77fee5ee5e3d5a82f276754b5bfa1fb66252a4 not found: ID does not exist" Oct 30 03:03:56.632883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.632853 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:56.634960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.634933 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-oauth-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.634962 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.634978 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/978d60b4-621f-4c30-bb10-ddae7ab03ffa-kube-api-access-b5k9g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.634990 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-oauth-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.635004 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-console-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.635017 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.635088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.635030 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978d60b4-621f-4c30-bb10-ddae7ab03ffa-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:03:56.646200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:56.646174 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5567f984fc-lhs5l"] Oct 30 03:03:58.150433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:03:58.150398 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" path="/var/lib/kubelet/pods/978d60b4-621f-4c30-bb10-ddae7ab03ffa/volumes" Oct 30 03:04:04.923630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:04.923592 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:04.972293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:04.972268 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:05.637438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:05.637392 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:07.622574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:07.622531 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-88c9b7875-l4m8f" podUID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" containerName="console" containerID="cri-o://b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f" gracePeriod=15 Oct 30 03:04:07.900827 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:07.900747 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:04:08.036174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036138 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036185 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036233 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036265 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\" (UniqueName: \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036294 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036330 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036375 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\") pod \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\" (UID: \"0fc4970a-26d2-4f7c-aaf9-597f1f308799\") " Oct 30 03:04:08.036707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036645 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:08.036759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036732 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:08.036860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036833 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:08.036918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.036898 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:08.038449 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.038420 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "kube-api-access-xl82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:08.038449 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.038438 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:08.038449 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.038422 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert" pod "0fc4970a-26d2-4f7c-aaf9-597f1f308799" (UID: "0fc4970a-26d2-4f7c-aaf9-597f1f308799"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:08.137928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137887 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-oauth-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.137928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137923 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.137928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137933 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.138165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137942 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fc4970a-26d2-4f7c-aaf9-597f1f308799-console-oauth-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.138165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137951 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl82v\" (UniqueName: \"kubernetes.io/projected/0fc4970a-26d2-4f7c-aaf9-597f1f308799-kube-api-access-xl82v\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.138165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137962 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fc4970a-26d2-4f7c-aaf9-597f1f308799-tmp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.138165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.137970 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc4970a-26d2-4f7c-aaf9-597f1f308799-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:08.632027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.631994 4560 generic.go:356] "Generic (PLEG): container finished" podID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" containerID="b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f" exitCode=0 Oct 30 03:04:08.632479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.632040 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c9b7875-l4m8f" Oct 30 03:04:08.632479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.632058 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c9b7875-l4m8f" event={"ID":"0fc4970a-26d2-4f7c-aaf9-597f1f308799","Type":"ContainerDied","Data":"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f"} Oct 30 03:04:08.632479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.632096 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c9b7875-l4m8f" event={"ID":"0fc4970a-26d2-4f7c-aaf9-597f1f308799","Type":"ContainerDied","Data":"a1b5a2adf3818eca329efcfaac258e41f2fe983dd7b8db5bd8741bc2543d5f51"} Oct 30 03:04:08.632479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.632115 4560 scope.go:122] "RemoveContainer" containerID="b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f" Oct 30 03:04:08.647601 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.647589 4560 scope.go:122] "RemoveContainer" containerID="b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f" Oct 30 03:04:08.647856 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:08.647839 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f\": container with ID starting with b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f not found: ID does not exist" containerID="b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f" Oct 30 03:04:08.647906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.647864 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f"} err="failed to get container status \"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f\": rpc error: code = NotFound desc = could not find container \"b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f\": container with ID starting with b570b0bc7edd937ce28634463055ea5f23a61fd7c14eb79ebd7c5305ec3f658f not found: ID does not exist" Oct 30 03:04:08.683644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.683599 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:04:08.696495 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:08.696465 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88c9b7875-l4m8f"] Oct 30 03:04:10.150302 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:10.150270 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" path="/var/lib/kubelet/pods/0fc4970a-26d2-4f7c-aaf9-597f1f308799/volumes" Oct 30 03:04:11.748123 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:11.748088 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:04:12.774999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:12.774961 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:04:12.775414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:12.775169 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" podUID="9dcd506e-f83a-47fe-a199-92a377ba3bca" containerName="insights-gathering" containerID="cri-o://51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1" gracePeriod=30 Oct 30 03:04:13.066842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.066815 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:04:13.074698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.074672 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") pod \"9dcd506e-f83a-47fe-a199-92a377ba3bca\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " Oct 30 03:04:13.074812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.074735 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") pod \"9dcd506e-f83a-47fe-a199-92a377ba3bca\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " Oct 30 03:04:13.074812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.074753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") pod \"9dcd506e-f83a-47fe-a199-92a377ba3bca\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " Oct 30 03:04:13.074891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.074856 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") pod \"9dcd506e-f83a-47fe-a199-92a377ba3bca\" (UID: \"9dcd506e-f83a-47fe-a199-92a377ba3bca\") " Oct 30 03:04:13.075059 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.075031 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle" pod "9dcd506e-f83a-47fe-a199-92a377ba3bca" (UID: "9dcd506e-f83a-47fe-a199-92a377ba3bca"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:13.075115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.075078 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path" pod "9dcd506e-f83a-47fe-a199-92a377ba3bca" (UID: "9dcd506e-f83a-47fe-a199-92a377ba3bca"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:13.075207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.075184 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle" pod "9dcd506e-f83a-47fe-a199-92a377ba3bca" (UID: "9dcd506e-f83a-47fe-a199-92a377ba3bca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:13.077057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.077041 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f" pod "9dcd506e-f83a-47fe-a199-92a377ba3bca" (UID: "9dcd506e-f83a-47fe-a199-92a377ba3bca"). InnerVolumeSpecName "kube-api-access-88r9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:13.176224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.176194 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-service-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:13.176224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.176223 4560 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9dcd506e-f83a-47fe-a199-92a377ba3bca-archives-path\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:13.176414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.176233 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88r9f\" (UniqueName: \"kubernetes.io/projected/9dcd506e-f83a-47fe-a199-92a377ba3bca-kube-api-access-88r9f\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:13.176414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.176241 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcd506e-f83a-47fe-a199-92a377ba3bca-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:13.655745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.655703 4560 generic.go:356] "Generic (PLEG): container finished" podID="9dcd506e-f83a-47fe-a199-92a377ba3bca" containerID="51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1" exitCode=2 Oct 30 03:04:13.655919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.655770 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" Oct 30 03:04:13.655919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.655792 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" event={"ID":"9dcd506e-f83a-47fe-a199-92a377ba3bca","Type":"ContainerDied","Data":"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1"} Oct 30 03:04:13.655919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.655876 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sjqrb-ntgv8" event={"ID":"9dcd506e-f83a-47fe-a199-92a377ba3bca","Type":"ContainerDied","Data":"a2279899889ea0e5707565633b690be252cbf346549757af568cc18fc55c1e7f"} Oct 30 03:04:13.655919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.655898 4560 scope.go:122] "RemoveContainer" containerID="51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1" Oct 30 03:04:13.666987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.666971 4560 scope.go:122] "RemoveContainer" containerID="51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1" Oct 30 03:04:13.667238 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:13.667217 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1\": container with ID starting with 51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1 not found: ID does not exist" containerID="51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1" Oct 30 03:04:13.667300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.667246 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1"} err="failed to get container status \"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1\": rpc error: code = NotFound desc = could not find container \"51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1\": container with ID starting with 51a77a452677c82390d57db3a4b70122fd655117c2ba5d4013fd67024150d4a1 not found: ID does not exist" Oct 30 03:04:13.708666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.708640 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:04:13.721405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:13.721377 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-sjqrb-ntgv8"] Oct 30 03:04:14.150632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:14.150593 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcd506e-f83a-47fe-a199-92a377ba3bca" path="/var/lib/kubelet/pods/9dcd506e-f83a-47fe-a199-92a377ba3bca/volumes" Oct 30 03:04:18.984018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.983985 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984279 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="alertmanager" containerID="cri-o://5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" gracePeriod=120 Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984349 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-metric" containerID="cri-o://6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" gracePeriod=120 Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984419 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="prom-label-proxy" containerID="cri-o://54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" gracePeriod=120 Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984430 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="config-reloader" containerID="cri-o://fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" gracePeriod=120 Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984413 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-web" containerID="cri-o://3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" gracePeriod=120 Oct 30 03:04:18.984622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:18.984423 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy" containerID="cri-o://4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" gracePeriod=120 Oct 30 03:04:19.686610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686577 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" exitCode=0 Oct 30 03:04:19.686610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686603 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" exitCode=0 Oct 30 03:04:19.686610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686609 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" exitCode=0 Oct 30 03:04:19.686610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686614 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" exitCode=0 Oct 30 03:04:19.686858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686648 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a"} Oct 30 03:04:19.686858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686737 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15"} Oct 30 03:04:19.686858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686765 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912"} Oct 30 03:04:19.686858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:19.686820 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2"} Oct 30 03:04:20.227917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.227893 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.336719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336636 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.336719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336690 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.336943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336725 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.336943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336764 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.336943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336900 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.336966 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337006 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337050 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337067 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337083 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337121 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337147 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.337367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337164 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\") pod \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\" (UID: \"89d6cd01-0918-4fa2-b6a3-066dd0e84984\") " Oct 30 03:04:20.338134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.337835 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:20.338264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.338146 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:20.338551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.338509 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:20.340195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.340166 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "kube-api-access-dnpl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:20.341708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.341668 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.341983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.341952 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.341983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.341962 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:20.342267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.342240 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.342371 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.342344 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.342655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.342629 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.343863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.343836 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:20.346030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.345923 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.353202 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.353175 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config" pod "89d6cd01-0918-4fa2-b6a3-066dd0e84984" (UID: "89d6cd01-0918-4fa2-b6a3-066dd0e84984"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:20.437793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437745 4560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-main-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.437793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437787 4560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.437793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437800 4560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-out\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.437793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437810 4560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437820 4560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-tls-assets\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437829 4560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-web-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437837 4560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437846 4560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-main-db\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437854 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-metrics-client-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437864 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpl4\" (UniqueName: \"kubernetes.io/projected/89d6cd01-0918-4fa2-b6a3-066dd0e84984-kube-api-access-dnpl4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437872 4560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d6cd01-0918-4fa2-b6a3-066dd0e84984-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437880 4560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-cluster-tls-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.438065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.437889 4560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89d6cd01-0918-4fa2-b6a3-066dd0e84984-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:20.694677 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694600 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" exitCode=0 Oct 30 03:04:20.694677 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694631 4560 generic.go:356] "Generic (PLEG): container finished" podID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerID="3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" exitCode=0 Oct 30 03:04:20.694857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694693 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.694857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694683 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106"} Oct 30 03:04:20.694857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694839 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee"} Oct 30 03:04:20.694949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694886 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89d6cd01-0918-4fa2-b6a3-066dd0e84984","Type":"ContainerDied","Data":"bd7cb339530f08e4b75397c52fba58ed37bddfa8ba3613c1f6fdfd9407ff1be7"} Oct 30 03:04:20.694949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.694900 4560 scope.go:122] "RemoveContainer" containerID="54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" Oct 30 03:04:20.702953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.702931 4560 scope.go:122] "RemoveContainer" containerID="6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" Oct 30 03:04:20.709894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.709877 4560 scope.go:122] "RemoveContainer" containerID="4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" Oct 30 03:04:20.716812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.716793 4560 scope.go:122] "RemoveContainer" containerID="3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" Oct 30 03:04:20.723173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.723158 4560 scope.go:122] "RemoveContainer" containerID="fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" Oct 30 03:04:20.730050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.730032 4560 scope.go:122] "RemoveContainer" containerID="5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" Oct 30 03:04:20.736894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.736877 4560 scope.go:122] "RemoveContainer" containerID="9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f" Oct 30 03:04:20.743638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.743620 4560 scope.go:122] "RemoveContainer" containerID="54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" Oct 30 03:04:20.743898 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.743881 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a\": container with ID starting with 54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a not found: ID does not exist" containerID="54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" Oct 30 03:04:20.743946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.743907 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a"} err="failed to get container status \"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a\": rpc error: code = NotFound desc = could not find container \"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a\": container with ID starting with 54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a not found: ID does not exist" Oct 30 03:04:20.743946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.743924 4560 scope.go:122] "RemoveContainer" containerID="6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" Oct 30 03:04:20.744159 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.744139 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106\": container with ID starting with 6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106 not found: ID does not exist" containerID="6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" Oct 30 03:04:20.744198 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744166 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106"} err="failed to get container status \"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106\": rpc error: code = NotFound desc = could not find container \"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106\": container with ID starting with 6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106 not found: ID does not exist" Oct 30 03:04:20.744198 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744182 4560 scope.go:122] "RemoveContainer" containerID="4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" Oct 30 03:04:20.744395 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.744378 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15\": container with ID starting with 4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15 not found: ID does not exist" containerID="4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" Oct 30 03:04:20.744442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744401 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15"} err="failed to get container status \"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15\": rpc error: code = NotFound desc = could not find container \"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15\": container with ID starting with 4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15 not found: ID does not exist" Oct 30 03:04:20.744442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744414 4560 scope.go:122] "RemoveContainer" containerID="3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" Oct 30 03:04:20.744670 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.744654 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee\": container with ID starting with 3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee not found: ID does not exist" containerID="3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" Oct 30 03:04:20.744715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744675 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee"} err="failed to get container status \"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee\": rpc error: code = NotFound desc = could not find container \"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee\": container with ID starting with 3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee not found: ID does not exist" Oct 30 03:04:20.744715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744689 4560 scope.go:122] "RemoveContainer" containerID="fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" Oct 30 03:04:20.744895 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.744880 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912\": container with ID starting with fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912 not found: ID does not exist" containerID="fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" Oct 30 03:04:20.744939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744898 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912"} err="failed to get container status \"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912\": rpc error: code = NotFound desc = could not find container \"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912\": container with ID starting with fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912 not found: ID does not exist" Oct 30 03:04:20.744939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.744910 4560 scope.go:122] "RemoveContainer" containerID="5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" Oct 30 03:04:20.745114 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.745098 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2\": container with ID starting with 5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2 not found: ID does not exist" containerID="5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" Oct 30 03:04:20.745148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745119 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2"} err="failed to get container status \"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2\": rpc error: code = NotFound desc = could not find container \"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2\": container with ID starting with 5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2 not found: ID does not exist" Oct 30 03:04:20.745148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745132 4560 scope.go:122] "RemoveContainer" containerID="9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f" Oct 30 03:04:20.745331 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:20.745317 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f\": container with ID starting with 9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f not found: ID does not exist" containerID="9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f" Oct 30 03:04:20.745369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745333 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f"} err="failed to get container status \"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f\": rpc error: code = NotFound desc = could not find container \"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f\": container with ID starting with 9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f not found: ID does not exist" Oct 30 03:04:20.745369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745345 4560 scope.go:122] "RemoveContainer" containerID="54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a" Oct 30 03:04:20.745593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745574 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a"} err="failed to get container status \"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a\": rpc error: code = NotFound desc = could not find container \"54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a\": container with ID starting with 54eb3d8a921cc1f246e522fa7b275721d842fef5b3a668f0ef107713d9c0103a not found: ID does not exist" Oct 30 03:04:20.745593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745593 4560 scope.go:122] "RemoveContainer" containerID="6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106" Oct 30 03:04:20.745813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745787 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106"} err="failed to get container status \"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106\": rpc error: code = NotFound desc = could not find container \"6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106\": container with ID starting with 6a21068905ea7ef8c9641fe2ec29f3d94cb614a9c650fb360b345350f5825106 not found: ID does not exist" Oct 30 03:04:20.745813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.745805 4560 scope.go:122] "RemoveContainer" containerID="4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15" Oct 30 03:04:20.746061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746042 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15"} err="failed to get container status \"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15\": rpc error: code = NotFound desc = could not find container \"4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15\": container with ID starting with 4064be31adac1c42b187806b020b00316b29dc0b8556aea77ab8c8a9aafb6c15 not found: ID does not exist" Oct 30 03:04:20.746132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746063 4560 scope.go:122] "RemoveContainer" containerID="3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee" Oct 30 03:04:20.746310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746288 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee"} err="failed to get container status \"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee\": rpc error: code = NotFound desc = could not find container \"3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee\": container with ID starting with 3474a6555ad4a6b793a19ba241da2c8146ec854e4201ecad4bb63fd53542b2ee not found: ID does not exist" Oct 30 03:04:20.746310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746307 4560 scope.go:122] "RemoveContainer" containerID="fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912" Oct 30 03:04:20.746504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746486 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912"} err="failed to get container status \"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912\": rpc error: code = NotFound desc = could not find container \"fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912\": container with ID starting with fec95e4b63ea670c1ce76734d4acea1417b5f88ee11beaa0efa69683f26a2912 not found: ID does not exist" Oct 30 03:04:20.746570 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746504 4560 scope.go:122] "RemoveContainer" containerID="5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2" Oct 30 03:04:20.746747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746726 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2"} err="failed to get container status \"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2\": rpc error: code = NotFound desc = could not find container \"5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2\": container with ID starting with 5cd2a0469d05b280d8d595ea6b57ba945b489507007de0edd080c91773c7a3a2 not found: ID does not exist" Oct 30 03:04:20.746821 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746748 4560 scope.go:122] "RemoveContainer" containerID="9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f" Oct 30 03:04:20.746978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.746960 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f"} err="failed to get container status \"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f\": rpc error: code = NotFound desc = could not find container \"9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f\": container with ID starting with 9b954102c9a19afdbc77de52c987f073bd56d35a4ac7cc6e52e31ec67aa57b8f not found: ID does not exist" Oct 30 03:04:20.752864 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.752842 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:20.765642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.765619 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:20.790281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790243 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:20.790657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790639 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="config-reloader" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790658 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="config-reloader" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790672 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-metric" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790679 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-metric" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790691 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="prom-label-proxy" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790698 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="prom-label-proxy" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790711 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-web" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790718 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-web" Oct 30 03:04:20.790750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790751 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9dcd506e-f83a-47fe-a199-92a377ba3bca" containerName="insights-gathering" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790758 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcd506e-f83a-47fe-a199-92a377ba3bca" containerName="insights-gathering" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790771 4560 cpu_manager.go:414] "containerMap: removing container" podUID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" containerName="console" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790778 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" containerName="console" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790802 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790809 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790821 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="init-config-reloader" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790828 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="init-config-reloader" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790843 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" containerName="console" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790850 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" containerName="console" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790860 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="alertmanager" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790867 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="alertmanager" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790943 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790957 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc4970a-26d2-4f7c-aaf9-597f1f308799" containerName="console" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790967 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="alertmanager" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790976 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="config-reloader" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790984 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-metric" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.790993 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="prom-label-proxy" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.791001 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" containerName="kube-rbac-proxy-web" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.791013 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcd506e-f83a-47fe-a199-92a377ba3bca" containerName="insights-gathering" Oct 30 03:04:20.791126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.791022 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="978d60b4-621f-4c30-bb10-ddae7ab03ffa" containerName="console" Oct 30 03:04:20.794828 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.794807 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.810886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810860 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810888 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810903 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810909 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810938 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810944 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810938 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810941 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fpvml\"" Oct 30 03:04:20.811010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.810968 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 03:04:20.823485 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.823459 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 03:04:20.828916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.828890 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:20.842380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842352 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842393 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842414 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvcq\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-kube-api-access-bmvcq\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842584 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842644 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842733 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-out\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842816 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-web-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.842947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842917 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.843032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842955 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-volume\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.843032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.842974 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.843032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.843021 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.843140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.843123 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943498 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943461 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943512 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943584 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943610 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvcq\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-kube-api-access-bmvcq\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943652 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943688 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943723 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943751 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-out\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-web-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943809 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943834 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-volume\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943858 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943892 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.943964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.943956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.944752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.944463 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.944752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.944550 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8551ab49-f11b-4823-bf2f-7b3a3c574c02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947070 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947114 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947158 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-volume\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947158 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947288 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8551ab49-f11b-4823-bf2f-7b3a3c574c02-config-out\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947346 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947490 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-web-config\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.947538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.947506 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.948610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.948587 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8551ab49-f11b-4823-bf2f-7b3a3c574c02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:20.965122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:20.965088 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvcq\" (UniqueName: \"kubernetes.io/projected/8551ab49-f11b-4823-bf2f-7b3a3c574c02-kube-api-access-bmvcq\") pod \"alertmanager-main-0\" (UID: \"8551ab49-f11b-4823-bf2f-7b3a3c574c02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:21.110845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:21.110810 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 03:04:21.448465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:21.448431 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 03:04:21.451377 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:04:21.451346 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8551ab49_f11b_4823_bf2f_7b3a3c574c02.slice/crio-08043d2b13f88679bbc8950898739a282b6e288ff3f1d0c74abed024f4925af4 WatchSource:0}: Error finding container 08043d2b13f88679bbc8950898739a282b6e288ff3f1d0c74abed024f4925af4: Status 404 returned error can't find the container with id 08043d2b13f88679bbc8950898739a282b6e288ff3f1d0c74abed024f4925af4 Oct 30 03:04:21.700752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:21.700661 4560 generic.go:356] "Generic (PLEG): container finished" podID="8551ab49-f11b-4823-bf2f-7b3a3c574c02" containerID="d2c624bf8226a2de7b65bc8ef430e2e0abba18e77782598643eaf0fdf7fab4c3" exitCode=0 Oct 30 03:04:21.700926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:21.700747 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerDied","Data":"d2c624bf8226a2de7b65bc8ef430e2e0abba18e77782598643eaf0fdf7fab4c3"} Oct 30 03:04:21.700926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:21.700867 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"08043d2b13f88679bbc8950898739a282b6e288ff3f1d0c74abed024f4925af4"} Oct 30 03:04:22.150253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.150220 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d6cd01-0918-4fa2-b6a3-066dd0e84984" path="/var/lib/kubelet/pods/89d6cd01-0918-4fa2-b6a3-066dd0e84984/volumes" Oct 30 03:04:22.710357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710318 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"eaa5628232cd2501f8dd22ebf23e18bf44fbe64f17bdfb90a8b1f873ee77e196"} Oct 30 03:04:22.710878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710397 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"61a39e3096c41f41497149fd047d9397f6ad7ce319cd9916698cab1feb1420b3"} Oct 30 03:04:22.710878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710441 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"bf43ec4ba6902e4d0c2ace9fe816e55f14bf5a7005d4e228ce77737a3654a2dd"} Oct 30 03:04:22.710878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710481 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"2ab8a61a5da517d21184e95b98a3fa4adddd3db7878e0d2f8e61ffbd8dafc6d8"} Oct 30 03:04:22.710878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710541 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"0fac48b67aa6b1fce5b75782375cb1c1a4751747be704e729509008feec06364"} Oct 30 03:04:22.710878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.710581 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8551ab49-f11b-4823-bf2f-7b3a3c574c02","Type":"ContainerStarted","Data":"2d63690dee21b348f951b19d138bdd1454bf41093430c73706400d07f9dfa91f"} Oct 30 03:04:22.788099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:22.788048 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.773811889 podStartE2EDuration="2.788034058s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.222169ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:22.787732582 +0000 UTC m=+90.772852471" watchObservedRunningTime="2026-10-30 03:04:22.788034058 +0000 UTC m=+90.773153947" Oct 30 03:04:23.574964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.574926 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575236 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="prometheus" containerID="cri-o://d9e9cde010559cc0c58ec3fe1d9a54189540908a89ee5f5816cb01763a6e01cd" gracePeriod=600 Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575246 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy" containerID="cri-o://9def3e44c02559f64c2b9c42b7699939a8178170c3c171c3e93b4ef1f3b5f837" gracePeriod=600 Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575275 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="thanos-sidecar" containerID="cri-o://af22b7692efedaf20fa38efee54ee95600fc723b913092ff9ed15c56efdf39a3" gracePeriod=600 Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575276 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://63069d3c76fd76a07c96deef9ebf93b7809bf1ac60319a832d36dc12ad057371" gracePeriod=600 Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575317 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-web" containerID="cri-o://137c2f366f7359de306f5acabbc66f4c614aecbc602144e26fc637a46b67599e" gracePeriod=600 Oct 30 03:04:23.575420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.575317 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="config-reloader" containerID="cri-o://c87105013ddb77760f41a91258d96f6b3f7ea9d8fd1d53a732b4e0dcc9578e73" gracePeriod=600 Oct 30 03:04:23.696580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.696553 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-746cf556c-g2m9k"] Oct 30 03:04:23.698598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.698581 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.717954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717927 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="63069d3c76fd76a07c96deef9ebf93b7809bf1ac60319a832d36dc12ad057371" exitCode=0 Oct 30 03:04:23.717954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717952 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="9def3e44c02559f64c2b9c42b7699939a8178170c3c171c3e93b4ef1f3b5f837" exitCode=0 Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717961 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="af22b7692efedaf20fa38efee54ee95600fc723b913092ff9ed15c56efdf39a3" exitCode=0 Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717967 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="c87105013ddb77760f41a91258d96f6b3f7ea9d8fd1d53a732b4e0dcc9578e73" exitCode=0 Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717975 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="d9e9cde010559cc0c58ec3fe1d9a54189540908a89ee5f5816cb01763a6e01cd" exitCode=0 Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.717995 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"63069d3c76fd76a07c96deef9ebf93b7809bf1ac60319a832d36dc12ad057371"} Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.718059 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"9def3e44c02559f64c2b9c42b7699939a8178170c3c171c3e93b4ef1f3b5f837"} Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.718097 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"af22b7692efedaf20fa38efee54ee95600fc723b913092ff9ed15c56efdf39a3"} Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.718188 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"c87105013ddb77760f41a91258d96f6b3f7ea9d8fd1d53a732b4e0dcc9578e73"} Oct 30 03:04:23.718307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.718228 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"d9e9cde010559cc0c58ec3fe1d9a54189540908a89ee5f5816cb01763a6e01cd"} Oct 30 03:04:23.720153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.720130 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746cf556c-g2m9k"] Oct 30 03:04:23.768540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768490 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.768694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768542 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpl9\" (UniqueName: \"kubernetes.io/projected/65b369dc-a5de-408f-941b-f19fa2124ebb-kube-api-access-dhpl9\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.768694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768649 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-console-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.768875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768852 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-service-ca\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.768939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768883 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b369dc-a5de-408f-941b-f19fa2124ebb-tmp\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.768939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.768918 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-trusted-ca-bundle\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.769205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.769190 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-oauth-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.769287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.769272 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-oauth-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870553 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-oauth-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870615 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-oauth-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870639 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870654 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpl9\" (UniqueName: \"kubernetes.io/projected/65b369dc-a5de-408f-941b-f19fa2124ebb-kube-api-access-dhpl9\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870672 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-console-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870699 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-service-ca\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870720 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b369dc-a5de-408f-941b-f19fa2124ebb-tmp\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.870967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.870834 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-trusted-ca-bundle\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.871479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.871453 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b369dc-a5de-408f-941b-f19fa2124ebb-tmp\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.871692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.871673 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-oauth-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.871767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.871675 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-service-ca\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.871809 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.871766 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-console-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.871957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.871940 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b369dc-a5de-408f-941b-f19fa2124ebb-trusted-ca-bundle\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.873080 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.873060 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-oauth-config\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.873165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.873151 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65b369dc-a5de-408f-941b-f19fa2124ebb-console-serving-cert\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:23.890430 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:23.890406 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpl9\" (UniqueName: \"kubernetes.io/projected/65b369dc-a5de-408f-941b-f19fa2124ebb-kube-api-access-dhpl9\") pod \"console-746cf556c-g2m9k\" (UID: \"65b369dc-a5de-408f-941b-f19fa2124ebb\") " pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:24.008223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.008187 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:24.305557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.305528 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746cf556c-g2m9k"] Oct 30 03:04:24.307389 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:04:24.307362 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b369dc_a5de_408f_941b_f19fa2124ebb.slice/crio-ba3d04739dfd91f0ca32c329da39bd68198113e347ef0a3587b68dc20a86dfd0 WatchSource:0}: Error finding container ba3d04739dfd91f0ca32c329da39bd68198113e347ef0a3587b68dc20a86dfd0: Status 404 returned error can't find the container with id ba3d04739dfd91f0ca32c329da39bd68198113e347ef0a3587b68dc20a86dfd0 Oct 30 03:04:24.725761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.725714 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746cf556c-g2m9k" event={"ID":"65b369dc-a5de-408f-941b-f19fa2124ebb","Type":"ContainerStarted","Data":"30d7568e27d74fc6727fe1908224a21fa89a733d5c8448297d3c4ed6b7089dfc"} Oct 30 03:04:24.726196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.725781 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746cf556c-g2m9k" event={"ID":"65b369dc-a5de-408f-941b-f19fa2124ebb","Type":"ContainerStarted","Data":"ba3d04739dfd91f0ca32c329da39bd68198113e347ef0a3587b68dc20a86dfd0"} Oct 30 03:04:24.729604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.729571 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerID="137c2f366f7359de306f5acabbc66f4c614aecbc602144e26fc637a46b67599e" exitCode=0 Oct 30 03:04:24.729735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.729612 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"137c2f366f7359de306f5acabbc66f4c614aecbc602144e26fc637a46b67599e"} Oct 30 03:04:24.769340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.769290 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-746cf556c-g2m9k" podStartSLOduration=1.7692724549999999 podStartE2EDuration="1.769272455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:24.768438744 +0000 UTC m=+92.753558655" watchObservedRunningTime="2026-10-30 03:04:24.769272455 +0000 UTC m=+92.754392337" Oct 30 03:04:24.827143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.827116 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:24.880409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880376 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880418 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880440 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880474 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880512 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880563 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880595 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880718 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880759 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880798 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.880853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880820 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880864 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880872 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880949 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.880979 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881012 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881014 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881039 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881097 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881129 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881310 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881344 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881366 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.881533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881375 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\") pod \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\" (UID: \"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c\") " Oct 30 03:04:24.881795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881700 4560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.881795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881721 4560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-metrics-client-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.881795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881735 4560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.881795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881750 4560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.882010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.881978 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.882650 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.882551 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.882886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.882861 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:24.883471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.883446 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.884474 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.884398 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.884592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.884482 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "kube-api-access-l8xw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:24.884919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.884892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:04:24.885237 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.885202 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.885435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.885377 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.885435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.885380 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:24.885435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.885396 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.885435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.885403 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:24.886222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.886184 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.886553 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.886534 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.886688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.886674 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.896974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.896949 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config" pod "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" (UID: "2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:24.982835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982803 4560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.982835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982833 4560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982847 4560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-db\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982859 4560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-kube-rbac-proxy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982872 4560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-grpc-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982883 4560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-metrics-client-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982894 4560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982908 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xw2\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-kube-api-access-l8xw2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982921 4560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-tls-assets\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982931 4560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config-out\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982941 4560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982954 4560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982965 4560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-web-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982977 4560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.982989 4560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:24.983066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:24.983003 4560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:04:25.737095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.737070 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.737095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.737075 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c","Type":"ContainerDied","Data":"0ea04e6c4e2e06dba0dccefc230816b4e015496d747a2c27e6b5d950279622c7"} Oct 30 03:04:25.737604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.737165 4560 scope.go:122] "RemoveContainer" containerID="63069d3c76fd76a07c96deef9ebf93b7809bf1ac60319a832d36dc12ad057371" Oct 30 03:04:25.744616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.744597 4560 scope.go:122] "RemoveContainer" containerID="9def3e44c02559f64c2b9c42b7699939a8178170c3c171c3e93b4ef1f3b5f837" Oct 30 03:04:25.751635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.751497 4560 scope.go:122] "RemoveContainer" containerID="137c2f366f7359de306f5acabbc66f4c614aecbc602144e26fc637a46b67599e" Oct 30 03:04:25.758110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.758095 4560 scope.go:122] "RemoveContainer" containerID="af22b7692efedaf20fa38efee54ee95600fc723b913092ff9ed15c56efdf39a3" Oct 30 03:04:25.764475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.764458 4560 scope.go:122] "RemoveContainer" containerID="c87105013ddb77760f41a91258d96f6b3f7ea9d8fd1d53a732b4e0dcc9578e73" Oct 30 03:04:25.771025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.771007 4560 scope.go:122] "RemoveContainer" containerID="d9e9cde010559cc0c58ec3fe1d9a54189540908a89ee5f5816cb01763a6e01cd" Oct 30 03:04:25.778090 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.778071 4560 scope.go:122] "RemoveContainer" containerID="0609b3d051804e4f8d1e10fee33c5758f54c6395b442f32baa2fecbafcb92da0" Oct 30 03:04:25.820225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.820197 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:25.833445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.833387 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:25.860613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860551 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:25.860906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860890 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860906 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860919 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-thanos" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860929 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-thanos" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860942 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="thanos-sidecar" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860947 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="thanos-sidecar" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860954 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-web" Oct 30 03:04:25.860971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860958 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-web" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860977 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="prometheus" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860982 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="prometheus" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860993 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="init-config-reloader" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.860998 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="init-config-reloader" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861004 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="config-reloader" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861009 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="config-reloader" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861062 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861072 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-thanos" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861080 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="thanos-sidecar" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861107 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="kube-rbac-proxy-web" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861122 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="prometheus" Oct 30 03:04:25.861230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.861136 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" containerName="config-reloader" Oct 30 03:04:25.866362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.866344 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.883006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.882975 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 03:04:25.883006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.882976 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 03:04:25.883137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.882982 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 03:04:25.883221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883205 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 03:04:25.883273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883227 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 03:04:25.883273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883269 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 03:04:25.883351 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883340 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 03:04:25.883437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883380 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 03:04:25.883482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883449 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 03:04:25.883606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883588 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 03:04:25.883679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883617 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 03:04:25.883679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883622 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-54ggx\"" Oct 30 03:04:25.883782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883694 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 03:04:25.883831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883778 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 03:04:25.883862 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.883847 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aq15m7t422gg8\"" Oct 30 03:04:25.898968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.897098 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 03:04:25.898968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.897121 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 03:04:25.898968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.898879 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:25.991928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.991892 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.991928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.991935 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.991967 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992049 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992090 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmjz\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-kube-api-access-4qmjz\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992110 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992179 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992208 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992232 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992293 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992347 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992375 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992395 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992418 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992473 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992490 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992509 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992555 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992660 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:25.992797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:25.992692 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093660 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093709 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093735 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093767 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093798 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093825 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.093951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093856 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093965 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.093995 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094035 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094063 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094093 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094121 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094158 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094183 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094223 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094256 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094285 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094314 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094359 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmjz\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-kube-api-access-4qmjz\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094501 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094544 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.094948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.094907 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.095238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.095212 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.095493 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.095445 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.095799 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.095775 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.097172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.097141 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.097336 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.097309 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.097948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.097610 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.097948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.097702 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.098099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.097951 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.098571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.098489 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.098727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.098668 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.098727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.098683 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf31f133-8a68-4de1-ad78-cf47f4159c32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.099321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.099296 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.099423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.099399 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf31f133-8a68-4de1-ad78-cf47f4159c32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.099684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.099664 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.099960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.099939 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-config\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.100318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.100303 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cf31f133-8a68-4de1-ad78-cf47f4159c32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.115220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.115169 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmjz\" (UniqueName: \"kubernetes.io/projected/cf31f133-8a68-4de1-ad78-cf47f4159c32-kube-api-access-4qmjz\") pod \"prometheus-k8s-0\" (UID: \"cf31f133-8a68-4de1-ad78-cf47f4159c32\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.149668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.149640 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c" path="/var/lib/kubelet/pods/2e89ec78-0ea0-4799-b7ac-cfc16c2a4a3c/volumes" Oct 30 03:04:26.191131 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.191101 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:26.525641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.525613 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:04:26.527118 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:04:26.527082 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf31f133_8a68_4de1_ad78_cf47f4159c32.slice/crio-785cb9ad78aabeab3da3c3b24bc1ccd27f0e33d0e3e3a69ea47e8f70304340b5 WatchSource:0}: Error finding container 785cb9ad78aabeab3da3c3b24bc1ccd27f0e33d0e3e3a69ea47e8f70304340b5: Status 404 returned error can't find the container with id 785cb9ad78aabeab3da3c3b24bc1ccd27f0e33d0e3e3a69ea47e8f70304340b5 Oct 30 03:04:26.744024 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.743987 4560 generic.go:356] "Generic (PLEG): container finished" podID="cf31f133-8a68-4de1-ad78-cf47f4159c32" containerID="571ae2d7b4dcc53c19760ad1cc4c357eeb2fe76daf3fae991c8f5b6ea62dc731" exitCode=0 Oct 30 03:04:26.744513 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.744042 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerDied","Data":"571ae2d7b4dcc53c19760ad1cc4c357eeb2fe76daf3fae991c8f5b6ea62dc731"} Oct 30 03:04:26.744513 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:26.744108 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"785cb9ad78aabeab3da3c3b24bc1ccd27f0e33d0e3e3a69ea47e8f70304340b5"} Oct 30 03:04:27.752199 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752158 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"ca0ecb43d917f586e5dbadc828af6f3ffbf16a26b112b4f95d22dc04f26c16a7"} Oct 30 03:04:27.752574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752279 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"33ac85070a636e08110de12149954001c23ebf67bd8a32ea521d3ef9758ad25a"} Oct 30 03:04:27.752574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752319 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"08fb52cca099198e2cfe5ecf10d014d7710623f4b796ce954596e0bc39e40c78"} Oct 30 03:04:27.752574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752355 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"c1bc93de8a8d1140dc9bd9f04cc6f32ca29f33fc74b42394e373da7d75724a9a"} Oct 30 03:04:27.752574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752391 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"046802d958a2362e21011258bf501edada2a59c0987570319f0b3e1f95e3454f"} Oct 30 03:04:27.752574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.752423 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cf31f133-8a68-4de1-ad78-cf47f4159c32","Type":"ContainerStarted","Data":"1b56ebcc9a5ce18bfcb85b115171c8c8e08149ba148ee29faf09c5a7a099f1c9"} Oct 30 03:04:27.818387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:27.818335 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.800184258 podStartE2EDuration="2.818320722s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.136464ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:27.816903431 +0000 UTC m=+95.802023344" watchObservedRunningTime="2026-10-30 03:04:27.818320722 +0000 UTC m=+95.803440636" Oct 30 03:04:31.191822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:31.191784 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:04:34.008802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:34.008763 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:34.009266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:34.008831 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:34.014884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:34.014850 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:34.787231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:34.787198 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-746cf556c-g2m9k" Oct 30 03:04:34.855666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:34.855635 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:04:41.748639 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:04:41.748598 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:04:59.877501 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:04:59.877427 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c4794fd9-kk5nx" podUID="164397bb-bbb8-4eab-8391-d31aab597329" containerName="console" containerID="cri-o://d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04" gracePeriod=15 Oct 30 03:05:00.134478 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.134423 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:05:00.327072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327033 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327101 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327128 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327151 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327223 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\" (UniqueName: \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327274 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327312 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\" (UniqueName: \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327371 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\") pod \"164397bb-bbb8-4eab-8391-d31aab597329\" (UID: \"164397bb-bbb8-4eab-8391-d31aab597329\") " Oct 30 03:05:00.327488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:00.327713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327586 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:05:00.327713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327675 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:00.327713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327681 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164397bb-bbb8-4eab-8391-d31aab597329-tmp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.327713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327699 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:00.327922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327718 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-console-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.327981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.327941 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:00.329668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.329645 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "kube-api-access-9sr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:00.329918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.329886 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:00.329967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.329914 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config" pod "164397bb-bbb8-4eab-8391-d31aab597329" (UID: "164397bb-bbb8-4eab-8391-d31aab597329"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428348 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-oauth-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428390 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/164397bb-bbb8-4eab-8391-d31aab597329-console-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428400 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428411 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-oauth-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428420 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr96\" (UniqueName: \"kubernetes.io/projected/164397bb-bbb8-4eab-8391-d31aab597329-kube-api-access-9sr96\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.428442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.428431 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/164397bb-bbb8-4eab-8391-d31aab597329-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:05:00.889291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.889259 4560 generic.go:356] "Generic (PLEG): container finished" podID="164397bb-bbb8-4eab-8391-d31aab597329" containerID="d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04" exitCode=0 Oct 30 03:05:00.889732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.889304 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4794fd9-kk5nx" Oct 30 03:05:00.889732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.889344 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4794fd9-kk5nx" event={"ID":"164397bb-bbb8-4eab-8391-d31aab597329","Type":"ContainerDied","Data":"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04"} Oct 30 03:05:00.889732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.889395 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4794fd9-kk5nx" event={"ID":"164397bb-bbb8-4eab-8391-d31aab597329","Type":"ContainerDied","Data":"00c00a56327fa9d3b1a1e49fc32c790cfdd8dc871f6ca289bcdd3ab70159e43e"} Oct 30 03:05:00.889732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.889417 4560 scope.go:122] "RemoveContainer" containerID="d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04" Oct 30 03:05:00.898020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.898005 4560 scope.go:122] "RemoveContainer" containerID="d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04" Oct 30 03:05:00.898298 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:05:00.898280 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04\": container with ID starting with d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04 not found: ID does not exist" containerID="d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04" Oct 30 03:05:00.898347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.898308 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04"} err="failed to get container status \"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04\": rpc error: code = NotFound desc = could not find container \"d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04\": container with ID starting with d5061ead811740ea76c0e7a514f5382c16ae8c9932106957bf8729efc4005b04 not found: ID does not exist" Oct 30 03:05:00.942390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.942356 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:05:00.955291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:00.955251 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c4794fd9-kk5nx"] Oct 30 03:05:02.149786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:02.149744 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164397bb-bbb8-4eab-8391-d31aab597329" path="/var/lib/kubelet/pods/164397bb-bbb8-4eab-8391-d31aab597329/volumes" Oct 30 03:05:11.747811 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:05:11.747717 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:05:26.191380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:26.191338 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:05:26.206652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:26.206628 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:05:27.008191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:05:27.008162 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:05:41.748345 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:05:41.748300 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:06:11.748054 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:06:11.748010 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:06:41.748198 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:06:41.748160 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:07:04.060613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.060575 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn"] Oct 30 03:07:04.061083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.060928 4560 cpu_manager.go:414] "containerMap: removing container" podUID="164397bb-bbb8-4eab-8391-d31aab597329" containerName="console" Oct 30 03:07:04.061083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.060939 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164397bb-bbb8-4eab-8391-d31aab597329" containerName="console" Oct 30 03:07:04.061083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.061002 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="164397bb-bbb8-4eab-8391-d31aab597329" containerName="console" Oct 30 03:07:04.063974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.063954 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.100824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.100788 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn"] Oct 30 03:07:04.149265 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.149235 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 03:07:04.149265 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.149230 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 03:07:04.151868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.151830 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 03:07:04.151868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.151857 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 03:07:04.152030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.151887 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 03:07:04.152030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.151914 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2t4j2\"" Oct 30 03:07:04.171868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.171840 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gjq\" (UniqueName: \"kubernetes.io/projected/27d3c4d1-8193-4117-b8dc-11962ca3d315-kube-api-access-s4gjq\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.171976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.171876 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.171976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.171906 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.172044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.172015 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.172082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.172053 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.172157 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.172143 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272542 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272588 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272626 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272692 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272736 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gjq\" (UniqueName: \"kubernetes.io/projected/27d3c4d1-8193-4117-b8dc-11962ca3d315-kube-api-access-s4gjq\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272988 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272768 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.272988 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.272977 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.275095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.275060 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d3c4d1-8193-4117-b8dc-11962ca3d315-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.275229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.275174 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.275229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.275201 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.275413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.275394 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d3c4d1-8193-4117-b8dc-11962ca3d315-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.293576 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.293550 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gjq\" (UniqueName: \"kubernetes.io/projected/27d3c4d1-8193-4117-b8dc-11962ca3d315-kube-api-access-s4gjq\") pod \"opendatahub-operator-controller-manager-5c85558465-z9pjn\" (UID: \"27d3c4d1-8193-4117-b8dc-11962ca3d315\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.378043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.377966 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:04.762587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:04.762555 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn"] Oct 30 03:07:04.780016 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:04.779991 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d3c4d1_8193_4117_b8dc_11962ca3d315.slice/crio-f7ebaad2e5cb04c4dcb2e4bc5f3886407000371984516edd58129365bdac0504 WatchSource:0}: Error finding container f7ebaad2e5cb04c4dcb2e4bc5f3886407000371984516edd58129365bdac0504: Status 404 returned error can't find the container with id f7ebaad2e5cb04c4dcb2e4bc5f3886407000371984516edd58129365bdac0504 Oct 30 03:07:05.366643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:05.366597 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" event={"ID":"27d3c4d1-8193-4117-b8dc-11962ca3d315","Type":"ContainerStarted","Data":"f7ebaad2e5cb04c4dcb2e4bc5f3886407000371984516edd58129365bdac0504"} Oct 30 03:07:08.383868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:08.383756 4560 generic.go:356] "Generic (PLEG): container finished" podID="27d3c4d1-8193-4117-b8dc-11962ca3d315" containerID="f17cfeba9f8d33399a28c30d0c925567269062b5b5ea7c60ca81687580c1f4b6" exitCode=0 Oct 30 03:07:08.383868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:08.383809 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" event={"ID":"27d3c4d1-8193-4117-b8dc-11962ca3d315","Type":"ContainerDied","Data":"f17cfeba9f8d33399a28c30d0c925567269062b5b5ea7c60ca81687580c1f4b6"} Oct 30 03:07:09.390620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:09.390585 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" event={"ID":"27d3c4d1-8193-4117-b8dc-11962ca3d315","Type":"ContainerStarted","Data":"aad6ad17ad78478025da8bc488a08c9b76ce05769c03654ad466002ffb52c204"} Oct 30 03:07:09.391073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:09.390708 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:09.473732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:09.473684 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" podStartSLOduration=2.858746608 podStartE2EDuration="6.473670569s" totalImagesPullingTime="3.133774612s" totalInitContainerRuntime="481.149349ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:09.45984657 +0000 UTC m=+257.444966463" watchObservedRunningTime="2026-10-30 03:07:09.473670569 +0000 UTC m=+257.458790456" Oct 30 03:07:11.747877 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:07:11.747730 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:07:12.806074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.806040 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz"] Oct 30 03:07:12.809584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.809564 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.838482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.838458 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:07:12.838482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.838462 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-nscsh\"" Oct 30 03:07:12.838653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.838502 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 03:07:12.840827 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.840803 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz"] Oct 30 03:07:12.852728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.852694 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-tmp\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.852841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.852806 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zr4\" (UniqueName: \"kubernetes.io/projected/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-kube-api-access-s7zr4\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.953907 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.953868 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-tmp\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.954086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.953933 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zr4\" (UniqueName: \"kubernetes.io/projected/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-kube-api-access-s7zr4\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.954295 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.954276 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-tmp\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:12.973577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:12.973551 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zr4\" (UniqueName: \"kubernetes.io/projected/589014f0-6bdc-4cc0-a96c-bf50fa44e36f-kube-api-access-s7zr4\") pod \"jobset-operator-674cd59dc5-vhbzz\" (UID: \"589014f0-6bdc-4cc0-a96c-bf50fa44e36f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:13.132312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:13.132206 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" Oct 30 03:07:13.435322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:13.435297 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz"] Oct 30 03:07:13.438277 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:13.438253 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod589014f0_6bdc_4cc0_a96c_bf50fa44e36f.slice/crio-b337106f052addd37845599d3cec7fac5c111bcc38e10ee17061fec65b75f7b9 WatchSource:0}: Error finding container b337106f052addd37845599d3cec7fac5c111bcc38e10ee17061fec65b75f7b9: Status 404 returned error can't find the container with id b337106f052addd37845599d3cec7fac5c111bcc38e10ee17061fec65b75f7b9 Oct 30 03:07:14.416926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:14.416886 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" event={"ID":"589014f0-6bdc-4cc0-a96c-bf50fa44e36f","Type":"ContainerStarted","Data":"b337106f052addd37845599d3cec7fac5c111bcc38e10ee17061fec65b75f7b9"} Oct 30 03:07:16.427291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:16.427254 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" event={"ID":"589014f0-6bdc-4cc0-a96c-bf50fa44e36f","Type":"ContainerStarted","Data":"c6d225886d5d4a977b1096865938bc2cf6c906d2451a9cffa0e44bd2a000e17b"} Oct 30 03:07:16.471709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:16.471657 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vhbzz" podStartSLOduration=1.625048091 podStartE2EDuration="4.471642813s" totalImagesPullingTime="2.846594722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:16.47042609 +0000 UTC m=+264.455545980" watchObservedRunningTime="2026-10-30 03:07:16.471642813 +0000 UTC m=+264.456762701" Oct 30 03:07:20.398411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:20.398382 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-z9pjn" Oct 30 03:07:30.683896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.683866 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb"] Oct 30 03:07:30.691338 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.691318 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.728450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.728412 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 03:07:30.728719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.728693 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-chwp9\"" Oct 30 03:07:30.728836 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.728735 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 03:07:30.732367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.732335 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb"] Oct 30 03:07:30.821619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.821581 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.821792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.821647 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlzz\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-kube-api-access-njlzz\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.922585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.922539 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlzz\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-kube-api-access-njlzz\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.922761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.922646 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.943323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.943249 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:30.943644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:30.943623 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlzz\" (UniqueName: \"kubernetes.io/projected/de986a53-a719-4883-b12b-79aa572c9272-kube-api-access-njlzz\") pod \"cert-manager-webhook-5dbbbf75cc-vfbtb\" (UID: \"de986a53-a719-4883-b12b-79aa572c9272\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:31.002630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:31.002595 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:31.322385 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:31.322351 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde986a53_a719_4883_b12b_79aa572c9272.slice/crio-055e38aed58f1bf244a27acb3f092b3f38452436d1b8d79046eb7097f729b710 WatchSource:0}: Error finding container 055e38aed58f1bf244a27acb3f092b3f38452436d1b8d79046eb7097f729b710: Status 404 returned error can't find the container with id 055e38aed58f1bf244a27acb3f092b3f38452436d1b8d79046eb7097f729b710 Oct 30 03:07:31.326746 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:31.326723 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb"] Oct 30 03:07:31.487775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:31.487741 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" event={"ID":"de986a53-a719-4883-b12b-79aa572c9272","Type":"ContainerStarted","Data":"055e38aed58f1bf244a27acb3f092b3f38452436d1b8d79046eb7097f729b710"} Oct 30 03:07:32.882488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:32.882430 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm"] Oct 30 03:07:32.886344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:32.886319 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:32.909224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:32.909041 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7b7sm\"" Oct 30 03:07:32.918801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:32.918767 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm"] Oct 30 03:07:33.046946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.046902 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdhc\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-kube-api-access-rrdhc\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.047103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.046961 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.148569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.148495 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdhc\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-kube-api-access-rrdhc\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.148569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.148560 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.169099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.169067 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.169722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.169697 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdhc\" (UniqueName: \"kubernetes.io/projected/ba30bd88-fb2b-471f-8acf-543ed6b14b20-kube-api-access-rrdhc\") pod \"cert-manager-cainjector-5c57bbd4fc-krlqm\" (UID: \"ba30bd88-fb2b-471f-8acf-543ed6b14b20\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.201537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.201487 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" Oct 30 03:07:33.531803 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:33.531666 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm"] Oct 30 03:07:33.534530 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:33.534485 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba30bd88_fb2b_471f_8acf_543ed6b14b20.slice/crio-ef78ad9acb70982c300d7f24f319fdf6bd96d02f32277ac2c226fbc11419fc59 WatchSource:0}: Error finding container ef78ad9acb70982c300d7f24f319fdf6bd96d02f32277ac2c226fbc11419fc59: Status 404 returned error can't find the container with id ef78ad9acb70982c300d7f24f319fdf6bd96d02f32277ac2c226fbc11419fc59 Oct 30 03:07:34.505343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:34.505302 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" event={"ID":"ba30bd88-fb2b-471f-8acf-543ed6b14b20","Type":"ContainerStarted","Data":"ef78ad9acb70982c300d7f24f319fdf6bd96d02f32277ac2c226fbc11419fc59"} Oct 30 03:07:35.511443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:35.511404 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" event={"ID":"de986a53-a719-4883-b12b-79aa572c9272","Type":"ContainerStarted","Data":"95775f65561c4f74995fea554ea83cd9de210cd6801e7ee8d31d7f34ae59b2f7"} Oct 30 03:07:35.511977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:35.511560 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:35.513457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:35.513430 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" event={"ID":"ba30bd88-fb2b-471f-8acf-543ed6b14b20","Type":"ContainerStarted","Data":"4b3a0822763411d8f740925a0ce0f3eece01c3f9d96bebef25325f5b97ec5e81"} Oct 30 03:07:35.564577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:35.561982 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" podStartSLOduration=2.105237376 podStartE2EDuration="5.561961432s" totalImagesPullingTime="3.456724056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:35.559573378 +0000 UTC m=+283.544693265" watchObservedRunningTime="2026-10-30 03:07:35.561961432 +0000 UTC m=+283.547081324" Oct 30 03:07:35.601643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:35.601587 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-krlqm" podStartSLOduration=1.764396705 podStartE2EDuration="3.601571746s" totalImagesPullingTime="1.837175041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:35.600125139 +0000 UTC m=+283.585245027" watchObservedRunningTime="2026-10-30 03:07:35.601571746 +0000 UTC m=+283.586691633" Oct 30 03:07:37.487177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.487133 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f"] Oct 30 03:07:37.490732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.490715 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.512400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.512375 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f"] Oct 30 03:07:37.522618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.522574 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 03:07:37.522618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.522596 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 03:07:37.522618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.522609 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 03:07:37.522855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.522601 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 03:07:37.522855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.522734 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-krtxf\"" Oct 30 03:07:37.593310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.593272 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tmp\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.593484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.593337 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjck\" (UniqueName: \"kubernetes.io/projected/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-kube-api-access-tgjck\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.593484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.593451 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.694362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.694323 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjck\" (UniqueName: \"kubernetes.io/projected/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-kube-api-access-tgjck\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.694555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.694407 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.694555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.694439 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tmp\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.696816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.696795 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tmp\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.697062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.697040 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.734379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.734348 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjck\" (UniqueName: \"kubernetes.io/projected/5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c-kube-api-access-tgjck\") pod \"kube-auth-proxy-78ccd95bcf-hzz4f\" (UID: \"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:37.804466 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:37.804387 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" Oct 30 03:07:38.132170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:38.132091 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f"] Oct 30 03:07:38.135920 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:38.135889 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b80a5d5_f055_4ab6_a5f7_0ea879f9a95c.slice/crio-cf9ed05afbd2208570db0a931b7312143ec472501e49eb005e2ddc428a2c5b16 WatchSource:0}: Error finding container cf9ed05afbd2208570db0a931b7312143ec472501e49eb005e2ddc428a2c5b16: Status 404 returned error can't find the container with id cf9ed05afbd2208570db0a931b7312143ec472501e49eb005e2ddc428a2c5b16 Oct 30 03:07:38.529041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:38.529001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" event={"ID":"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c","Type":"ContainerStarted","Data":"cf9ed05afbd2208570db0a931b7312143ec472501e49eb005e2ddc428a2c5b16"} Oct 30 03:07:39.106229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.106195 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:39.110079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.110047 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.127296 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.127272 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clchq\"" Oct 30 03:07:39.131626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.131598 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:39.139924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.139898 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 03:07:39.139924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.139926 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 03:07:39.140135 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.140114 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 03:07:39.208358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.208320 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rz4m\" (UniqueName: \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.208563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.208501 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.247458 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.247420 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:39.309817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.309778 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.309998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.309874 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rz4m\" (UniqueName: \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.310642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.310612 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.329959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.329924 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rz4m\" (UniqueName: \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\") pod \"limitador-limitador-74dfcfb894-x5j7h\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.427258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.427174 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:39.735801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:39.735764 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:39.737161 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:39.737128 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a1bbf3_3fc9_4b58_951b_0905fe4467f7.slice/crio-b1f905962a782897a0dcbc75e7af8aafcc241acaa654acaa7087a91c6af357ed WatchSource:0}: Error finding container b1f905962a782897a0dcbc75e7af8aafcc241acaa654acaa7087a91c6af357ed: Status 404 returned error can't find the container with id b1f905962a782897a0dcbc75e7af8aafcc241acaa654acaa7087a91c6af357ed Oct 30 03:07:40.542098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.542061 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" event={"ID":"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7","Type":"ContainerStarted","Data":"b1f905962a782897a0dcbc75e7af8aafcc241acaa654acaa7087a91c6af357ed"} Oct 30 03:07:40.609545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.609487 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:07:40.613088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.613060 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:07:40.624600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.624573 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr99x\" (UniqueName: \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\") pod \"authorino-56c6c9ddd5-nnm6r\" (UID: \"beb953e6-1c9c-4086-8001-a073478db015\") " pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:07:40.636055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.636023 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:07:40.726096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.726045 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr99x\" (UniqueName: \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\") pod \"authorino-56c6c9ddd5-nnm6r\" (UID: \"beb953e6-1c9c-4086-8001-a073478db015\") " pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:07:40.727949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.727927 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dd2cq\"" Oct 30 03:07:40.746471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.746446 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr99x\" (UniqueName: \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\") pod \"authorino-56c6c9ddd5-nnm6r\" (UID: \"beb953e6-1c9c-4086-8001-a073478db015\") " pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:07:40.926762 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:40.926675 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:07:41.234006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.233978 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:07:41.236359 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:41.236329 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb953e6_1c9c_4086_8001_a073478db015.slice/crio-8009e1079c3503f573dade63b30cacbbf4e838775692370a60eb9595895ffdc0 WatchSource:0}: Error finding container 8009e1079c3503f573dade63b30cacbbf4e838775692370a60eb9595895ffdc0: Status 404 returned error can't find the container with id 8009e1079c3503f573dade63b30cacbbf4e838775692370a60eb9595895ffdc0 Oct 30 03:07:41.521090 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.521020 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vfbtb" Oct 30 03:07:41.548854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.548821 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" event={"ID":"5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c","Type":"ContainerStarted","Data":"23a01c5dda5f75de66d601e6caf236859fbebbdfa321f44f6bc594ae2d7c099e"} Oct 30 03:07:41.550309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.550282 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" event={"ID":"beb953e6-1c9c-4086-8001-a073478db015","Type":"ContainerStarted","Data":"8009e1079c3503f573dade63b30cacbbf4e838775692370a60eb9595895ffdc0"} Oct 30 03:07:41.614075 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.614009 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-hzz4f" podStartSLOduration=2.129944112 podStartE2EDuration="4.613987581s" totalImagesPullingTime="2.484043469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:41.611186326 +0000 UTC m=+289.596306214" watchObservedRunningTime="2026-10-30 03:07:41.613987581 +0000 UTC m=+289.599107470" Oct 30 03:07:41.765326 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.765289 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm"] Oct 30 03:07:41.765757 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:07:41.765712 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:07:41.769865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.769840 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.785187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.785143 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-fzwlr\"" Oct 30 03:07:41.797545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.797487 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm"] Oct 30 03:07:41.839545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.839496 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrtz\" (UniqueName: \"kubernetes.io/projected/f1609387-b7a1-4e32-afce-6e893fb62865-kube-api-access-rlrtz\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.839734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.839707 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.839787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.839751 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1609387-b7a1-4e32-afce-6e893fb62865-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.839826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.839782 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.850021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.849781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 03:07:41.850021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.849962 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 03:07:41.863138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.863091 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 03:07:41.940867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.940828 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.941037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.940878 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1609387-b7a1-4e32-afce-6e893fb62865-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.941037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.940911 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.941037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.940981 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrtz\" (UniqueName: \"kubernetes.io/projected/f1609387-b7a1-4e32-afce-6e893fb62865-kube-api-access-rlrtz\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.941776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.941750 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.942214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.942192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1609387-b7a1-4e32-afce-6e893fb62865-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.943124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.943006 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1609387-b7a1-4e32-afce-6e893fb62865-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:41.962006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:41.961978 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrtz\" (UniqueName: \"kubernetes.io/projected/f1609387-b7a1-4e32-afce-6e893fb62865-kube-api-access-rlrtz\") pod \"kserve-module-controller-manager-7566cf5fc8-bp7bm\" (UID: \"f1609387-b7a1-4e32-afce-6e893fb62865\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:42.091056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:42.090976 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:42.468440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:42.468407 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm"] Oct 30 03:07:42.471245 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:07:42.471210 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1609387_b7a1_4e32_afce_6e893fb62865.slice/crio-fa57123e18c8441c6d4df20482cc4876fa7bcf6bce5f3cd5bbcf3f3a90545866 WatchSource:0}: Error finding container fa57123e18c8441c6d4df20482cc4876fa7bcf6bce5f3cd5bbcf3f3a90545866: Status 404 returned error can't find the container with id fa57123e18c8441c6d4df20482cc4876fa7bcf6bce5f3cd5bbcf3f3a90545866 Oct 30 03:07:42.557918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:42.557876 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" event={"ID":"f1609387-b7a1-4e32-afce-6e893fb62865","Type":"ContainerStarted","Data":"fa57123e18c8441c6d4df20482cc4876fa7bcf6bce5f3cd5bbcf3f3a90545866"} Oct 30 03:07:43.568548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:43.564664 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" event={"ID":"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7","Type":"ContainerStarted","Data":"d1abaf91fccd0624dff01378649c4af57e3fee29dd2b8c96ee4288e59fd2dc9e"} Oct 30 03:07:43.568548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:43.564741 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:43.568548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:43.566712 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" event={"ID":"beb953e6-1c9c-4086-8001-a073478db015","Type":"ContainerStarted","Data":"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667"} Oct 30 03:07:43.606337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:43.606278 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" podStartSLOduration=1.61647543 podStartE2EDuration="4.606260133s" totalImagesPullingTime="2.989784703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:43.605440382 +0000 UTC m=+291.590560271" watchObservedRunningTime="2026-10-30 03:07:43.606260133 +0000 UTC m=+291.591380020" Oct 30 03:07:43.646115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:43.646043 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" podStartSLOduration=1.756703075 podStartE2EDuration="3.646026653s" totalImagesPullingTime="1.889323578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:43.644689078 +0000 UTC m=+291.629808977" watchObservedRunningTime="2026-10-30 03:07:43.646026653 +0000 UTC m=+291.631146542" Oct 30 03:07:44.572851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:44.572801 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" event={"ID":"f1609387-b7a1-4e32-afce-6e893fb62865","Type":"ContainerStarted","Data":"9ee40b53ce5930544fe6bad5202f7ec2bc010c103c970927327cac914d1e4f83"} Oct 30 03:07:44.628541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:44.628465 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" podStartSLOduration=2.104885418 podStartE2EDuration="3.628445772s" totalImagesPullingTime="1.523560354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:44.627935157 +0000 UTC m=+292.613055058" watchObservedRunningTime="2026-10-30 03:07:44.628445772 +0000 UTC m=+292.613565661" Oct 30 03:07:45.577741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:45.577701 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:07:54.573847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:54.573817 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:55.204012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.203975 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:55.204191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.204169 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" podUID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" containerName="limitador" containerID="cri-o://d1abaf91fccd0624dff01378649c4af57e3fee29dd2b8c96ee4288e59fd2dc9e" gracePeriod=30 Oct 30 03:07:55.622752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.622724 4560 generic.go:356] "Generic (PLEG): container finished" podID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" containerID="d1abaf91fccd0624dff01378649c4af57e3fee29dd2b8c96ee4288e59fd2dc9e" exitCode=0 Oct 30 03:07:55.623048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.622788 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" event={"ID":"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7","Type":"ContainerDied","Data":"d1abaf91fccd0624dff01378649c4af57e3fee29dd2b8c96ee4288e59fd2dc9e"} Oct 30 03:07:55.746755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.746726 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:55.871003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.870922 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\" (UniqueName: \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\") pod \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " Oct 30 03:07:55.871003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.870976 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\" (UniqueName: \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\") pod \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\" (UID: \"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7\") " Oct 30 03:07:55.871263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.871240 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file" pod "e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" (UID: "e2a1bbf3-3fc9-4b58-951b-0905fe4467f7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:07:55.873125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.873101 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m" pod "e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" (UID: "e2a1bbf3-3fc9-4b58-951b-0905fe4467f7"). InnerVolumeSpecName "kube-api-access-8rz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:07:55.972292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.972262 4560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-config-file\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:07:55.972292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:55.972290 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rz4m\" (UniqueName: \"kubernetes.io/projected/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7-kube-api-access-8rz4m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:07:56.628528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:56.628490 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" Oct 30 03:07:56.628933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:56.628513 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x5j7h" event={"ID":"e2a1bbf3-3fc9-4b58-951b-0905fe4467f7","Type":"ContainerDied","Data":"b1f905962a782897a0dcbc75e7af8aafcc241acaa654acaa7087a91c6af357ed"} Oct 30 03:07:56.628933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:56.628587 4560 scope.go:122] "RemoveContainer" containerID="d1abaf91fccd0624dff01378649c4af57e3fee29dd2b8c96ee4288e59fd2dc9e" Oct 30 03:07:56.679213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:56.679176 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:56.691245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:56.691217 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x5j7h"] Oct 30 03:07:58.150789 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:07:58.150747 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" path="/var/lib/kubelet/pods/e2a1bbf3-3fc9-4b58-951b-0905fe4467f7/volumes" Oct 30 03:08:06.583722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:06.583643 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bp7bm" Oct 30 03:08:11.748470 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:08:11.748432 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:08:37.227973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.227936 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9qzhs"] Oct 30 03:08:37.228589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.228320 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" containerName="limitador" Oct 30 03:08:37.228589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.228333 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" containerName="limitador" Oct 30 03:08:37.228589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.228417 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a1bbf3-3fc9-4b58-951b-0905fe4467f7" containerName="limitador" Oct 30 03:08:37.233481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.233461 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.254128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.254092 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9qzhs"] Oct 30 03:08:37.264714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.264683 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-25xt9\"" Oct 30 03:08:37.264885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.264770 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 03:08:37.270319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.270294 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:08:37.273978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.273949 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.285578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.285551 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c02093-c650-44e2-a1b1-a8c41c98f0c2-cert\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.285682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.285638 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svvw\" (UniqueName: \"kubernetes.io/projected/92c02093-c650-44e2-a1b1-a8c41c98f0c2-kube-api-access-5svvw\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.295254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.295196 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:08:37.301061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.301020 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5jkcd\"" Oct 30 03:08:37.302251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.301095 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 03:08:37.316103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.316076 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-r2cdp"] Oct 30 03:08:37.319939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.319921 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.339040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.339014 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-r2cdp"] Oct 30 03:08:37.352505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.352480 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5ftmk\"" Oct 30 03:08:37.352645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.352551 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 03:08:37.386652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.386624 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.386652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.386662 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v7m\" (UniqueName: \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.386872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.386775 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c02093-c650-44e2-a1b1-a8c41c98f0c2-cert\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.386872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.386817 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3ee93-3d9d-4681-a38f-f745d04799ff-tls-certs\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.387018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.386999 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svvw\" (UniqueName: \"kubernetes.io/projected/92c02093-c650-44e2-a1b1-a8c41c98f0c2-kube-api-access-5svvw\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.387096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.387077 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4xv\" (UniqueName: \"kubernetes.io/projected/11c3ee93-3d9d-4681-a38f-f745d04799ff-kube-api-access-wf4xv\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.389256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.389235 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c02093-c650-44e2-a1b1-a8c41c98f0c2-cert\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.407424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.407380 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svvw\" (UniqueName: \"kubernetes.io/projected/92c02093-c650-44e2-a1b1-a8c41c98f0c2-kube-api-access-5svvw\") pod \"kserve-controller-manager-776cc66d47-9qzhs\" (UID: \"92c02093-c650-44e2-a1b1-a8c41c98f0c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.488511 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.488422 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3ee93-3d9d-4681-a38f-f745d04799ff-tls-certs\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.488714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.488604 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4xv\" (UniqueName: \"kubernetes.io/projected/11c3ee93-3d9d-4681-a38f-f745d04799ff-kube-api-access-wf4xv\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.488714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.488672 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.489071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.489023 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v7m\" (UniqueName: \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.491682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.491660 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.491780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.491705 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3ee93-3d9d-4681-a38f-f745d04799ff-tls-certs\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.508036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.508007 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v7m\" (UniqueName: \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\") pod \"llmisvc-controller-manager-869d5dcc5d-lp4cp\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.508639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.508618 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4xv\" (UniqueName: \"kubernetes.io/projected/11c3ee93-3d9d-4681-a38f-f745d04799ff-kube-api-access-wf4xv\") pod \"model-serving-api-75bd89fdfc-r2cdp\" (UID: \"11c3ee93-3d9d-4681-a38f-f745d04799ff\") " pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.547859 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.547832 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:37.587074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.587044 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:37.633595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.633561 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:37.847324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.847300 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9qzhs"] Oct 30 03:08:37.849757 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:08:37.849732 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c02093_c650_44e2_a1b1_a8c41c98f0c2.slice/crio-dd740dfb2b8ca4f6c794b94aa97662f3052f5488ecef7f3c5e0929a7e49ea518 WatchSource:0}: Error finding container dd740dfb2b8ca4f6c794b94aa97662f3052f5488ecef7f3c5e0929a7e49ea518: Status 404 returned error can't find the container with id dd740dfb2b8ca4f6c794b94aa97662f3052f5488ecef7f3c5e0929a7e49ea518 Oct 30 03:08:37.851127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.851112 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:08:37.900118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.900045 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:08:37.902281 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:08:37.902253 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc75b4ca1_ffa0_41c2_8244_d94da2ed3d7c.slice/crio-4101714bfcc86df5483b023d64247d20e4d89bd0d1af2ea3de27fc05e7d37f78 WatchSource:0}: Error finding container 4101714bfcc86df5483b023d64247d20e4d89bd0d1af2ea3de27fc05e7d37f78: Status 404 returned error can't find the container with id 4101714bfcc86df5483b023d64247d20e4d89bd0d1af2ea3de27fc05e7d37f78 Oct 30 03:08:37.946658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:37.946636 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-r2cdp"] Oct 30 03:08:37.948632 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:08:37.948609 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c3ee93_3d9d_4681_a38f_f745d04799ff.slice/crio-004ce287cc3db10295fed0e484273c235180c4ed080deca3a9f1c5cdd7107f93 WatchSource:0}: Error finding container 004ce287cc3db10295fed0e484273c235180c4ed080deca3a9f1c5cdd7107f93: Status 404 returned error can't find the container with id 004ce287cc3db10295fed0e484273c235180c4ed080deca3a9f1c5cdd7107f93 Oct 30 03:08:38.829619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:38.829582 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" event={"ID":"11c3ee93-3d9d-4681-a38f-f745d04799ff","Type":"ContainerStarted","Data":"004ce287cc3db10295fed0e484273c235180c4ed080deca3a9f1c5cdd7107f93"} Oct 30 03:08:38.831413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:38.831363 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" event={"ID":"92c02093-c650-44e2-a1b1-a8c41c98f0c2","Type":"ContainerStarted","Data":"dd740dfb2b8ca4f6c794b94aa97662f3052f5488ecef7f3c5e0929a7e49ea518"} Oct 30 03:08:38.833339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:38.833311 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" event={"ID":"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c","Type":"ContainerStarted","Data":"4101714bfcc86df5483b023d64247d20e4d89bd0d1af2ea3de27fc05e7d37f78"} Oct 30 03:08:40.847990 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.847951 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" event={"ID":"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c","Type":"ContainerStarted","Data":"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa"} Oct 30 03:08:40.848419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.848149 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:08:40.849692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.849663 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" event={"ID":"11c3ee93-3d9d-4681-a38f-f745d04799ff","Type":"ContainerStarted","Data":"41e549691f95be8935b69744f93b679815003aaa3d751dae9e32f63b18743ef5"} Oct 30 03:08:40.849842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.849721 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:08:40.851442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.851411 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" event={"ID":"92c02093-c650-44e2-a1b1-a8c41c98f0c2","Type":"ContainerStarted","Data":"ae792d0babb62d16cd2449133fb4fcda1d74a70b1793fb7ec66a6e2f14f176de"} Oct 30 03:08:40.851601 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.851493 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:08:40.890217 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.890157 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" podStartSLOduration=1.5167462029999998 podStartE2EDuration="3.890142838s" totalImagesPullingTime="2.373396635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:40.889386434 +0000 UTC m=+348.874506324" watchObservedRunningTime="2026-10-30 03:08:40.890142838 +0000 UTC m=+348.875262767" Oct 30 03:08:40.930812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.930762 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" podStartSLOduration=1.354899578 podStartE2EDuration="3.930747453s" totalImagesPullingTime="2.575847875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:40.929772178 +0000 UTC m=+348.914892062" watchObservedRunningTime="2026-10-30 03:08:40.930747453 +0000 UTC m=+348.915867341" Oct 30 03:08:40.971843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:40.971785 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" podStartSLOduration=2.067208156 podStartE2EDuration="3.971770771s" totalImagesPullingTime="1.904562615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:40.970211812 +0000 UTC m=+348.955331703" watchObservedRunningTime="2026-10-30 03:08:40.971770771 +0000 UTC m=+348.956890658" Oct 30 03:08:41.748421 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:08:41.748388 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:08:47.068459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.068418 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk"] Oct 30 03:08:47.072689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.072666 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.090021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.089996 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 03:08:47.104398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.104366 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk"] Oct 30 03:08:47.105584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.105560 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-d7rqg\"" Oct 30 03:08:47.105710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.105667 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 03:08:47.105919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.105903 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 03:08:47.184561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.184505 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d401e773-6801-4310-99f3-103d4435d93e-manager-config\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.184792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.184566 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-cert\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.184792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.184745 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-metrics-certs\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.184792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.184778 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2xp\" (UniqueName: \"kubernetes.io/projected/d401e773-6801-4310-99f3-103d4435d93e-kube-api-access-9k2xp\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.285898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.285855 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-metrics-certs\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.285898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.285900 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2xp\" (UniqueName: \"kubernetes.io/projected/d401e773-6801-4310-99f3-103d4435d93e-kube-api-access-9k2xp\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.286162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.285953 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d401e773-6801-4310-99f3-103d4435d93e-manager-config\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.286162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.285977 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-cert\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.286705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.286676 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d401e773-6801-4310-99f3-103d4435d93e-manager-config\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.288435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.288406 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-cert\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.288552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.288443 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401e773-6801-4310-99f3-103d4435d93e-metrics-certs\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.332284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.332200 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2xp\" (UniqueName: \"kubernetes.io/projected/d401e773-6801-4310-99f3-103d4435d93e-kube-api-access-9k2xp\") pod \"jobset-controller-manager-d954db9b5-dnclk\" (UID: \"d401e773-6801-4310-99f3-103d4435d93e\") " pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.387242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.387204 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:47.707645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.707616 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk"] Oct 30 03:08:47.710413 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:08:47.710383 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd401e773_6801_4310_99f3_103d4435d93e.slice/crio-ca601fb96afe106bd0a0b133f65310cc69d94fa0e2ce9ceedeeee3f0429d1745 WatchSource:0}: Error finding container ca601fb96afe106bd0a0b133f65310cc69d94fa0e2ce9ceedeeee3f0429d1745: Status 404 returned error can't find the container with id ca601fb96afe106bd0a0b133f65310cc69d94fa0e2ce9ceedeeee3f0429d1745 Oct 30 03:08:47.891096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:47.891055 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" event={"ID":"d401e773-6801-4310-99f3-103d4435d93e","Type":"ContainerStarted","Data":"ca601fb96afe106bd0a0b133f65310cc69d94fa0e2ce9ceedeeee3f0429d1745"} Oct 30 03:08:50.908252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:50.908217 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" event={"ID":"d401e773-6801-4310-99f3-103d4435d93e","Type":"ContainerStarted","Data":"99de46dec00942757592b4cf8071fe84c69391f4feb2c0cff37ac3fd315ca91c"} Oct 30 03:08:50.908652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:50.908361 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:08:50.961149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:50.961080 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" podStartSLOduration=1.273837677 podStartE2EDuration="3.961066137s" totalImagesPullingTime="2.68722846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:50.960960117 +0000 UTC m=+358.946080006" watchObservedRunningTime="2026-10-30 03:08:50.961066137 +0000 UTC m=+358.946186025" Oct 30 03:08:51.861949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:08:51.861917 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-r2cdp" Oct 30 03:09:01.918234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:01.918201 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-d954db9b5-dnclk" Oct 30 03:09:03.857873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:03.857831 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:09:03.858368 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:03.858068 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" podUID="beb953e6-1c9c-4086-8001-a073478db015" containerName="authorino" containerID="cri-o://e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667" gracePeriod=30 Oct 30 03:09:04.111060 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.111001 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:09:04.145790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.145760 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\" (UniqueName: \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\") pod \"beb953e6-1c9c-4086-8001-a073478db015\" (UID: \"beb953e6-1c9c-4086-8001-a073478db015\") " Oct 30 03:09:04.148197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.148140 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x" pod "beb953e6-1c9c-4086-8001-a073478db015" (UID: "beb953e6-1c9c-4086-8001-a073478db015"). InnerVolumeSpecName "kube-api-access-lr99x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:04.247098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.247066 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr99x\" (UniqueName: \"kubernetes.io/projected/beb953e6-1c9c-4086-8001-a073478db015-kube-api-access-lr99x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:09:04.976210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.976177 4560 generic.go:356] "Generic (PLEG): container finished" podID="beb953e6-1c9c-4086-8001-a073478db015" containerID="e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667" exitCode=0 Oct 30 03:09:04.976637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.976220 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" Oct 30 03:09:04.976637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.976256 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" event={"ID":"beb953e6-1c9c-4086-8001-a073478db015","Type":"ContainerDied","Data":"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667"} Oct 30 03:09:04.976637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.976287 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nnm6r" event={"ID":"beb953e6-1c9c-4086-8001-a073478db015","Type":"ContainerDied","Data":"8009e1079c3503f573dade63b30cacbbf4e838775692370a60eb9595895ffdc0"} Oct 30 03:09:04.976637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.976303 4560 scope.go:122] "RemoveContainer" containerID="e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667" Oct 30 03:09:04.984640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.984611 4560 scope.go:122] "RemoveContainer" containerID="e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667" Oct 30 03:09:04.984921 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:09:04.984900 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667\": container with ID starting with e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667 not found: ID does not exist" containerID="e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667" Oct 30 03:09:04.984974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:04.984930 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667"} err="failed to get container status \"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667\": rpc error: code = NotFound desc = could not find container \"e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667\": container with ID starting with e33068a82a240532508ed9ea44559d33be294f0ba11f81b61f3d29a3694d4667 not found: ID does not exist" Oct 30 03:09:05.026809 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:05.026777 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:09:05.032269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:05.032245 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nnm6r"] Oct 30 03:09:06.150327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:06.150296 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb953e6-1c9c-4086-8001-a073478db015" path="/var/lib/kubelet/pods/beb953e6-1c9c-4086-8001-a073478db015/volumes" Oct 30 03:09:11.748013 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:09:11.747975 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:09:11.858823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:11.858795 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:09:11.861857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:11.861828 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9qzhs" Oct 30 03:09:41.748797 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:09:41.748721 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:09:45.172779 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.172746 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jk86r"] Oct 30 03:09:45.173194 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.173167 4560 cpu_manager.go:414] "containerMap: removing container" podUID="beb953e6-1c9c-4086-8001-a073478db015" containerName="authorino" Oct 30 03:09:45.173194 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.173179 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb953e6-1c9c-4086-8001-a073478db015" containerName="authorino" Oct 30 03:09:45.173266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.173246 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb953e6-1c9c-4086-8001-a073478db015" containerName="authorino" Oct 30 03:09:45.175698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.175683 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.194980 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.194946 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jk86r"] Oct 30 03:09:45.221317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.221275 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-cert\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.221508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.221414 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrfr\" (UniqueName: \"kubernetes.io/projected/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-kube-api-access-ctrfr\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.322264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.322227 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrfr\" (UniqueName: \"kubernetes.io/projected/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-kube-api-access-ctrfr\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.322433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.322318 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-cert\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.324760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.324740 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-cert\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.341999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.341969 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrfr\" (UniqueName: \"kubernetes.io/projected/504a6d3e-f83e-4c3e-92a7-0db908fcfe00-kube-api-access-ctrfr\") pod \"llmisvc-controller-manager-6466c554d8-jk86r\" (UID: \"504a6d3e-f83e-4c3e-92a7-0db908fcfe00\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.488444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.488405 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:45.787181 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:45.787153 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jk86r"] Oct 30 03:09:45.789328 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:09:45.789297 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504a6d3e_f83e_4c3e_92a7_0db908fcfe00.slice/crio-b66fd0f5ef521d673dd4441d697e342b3996e85134d2a4d151ccef4bd60bed3f WatchSource:0}: Error finding container b66fd0f5ef521d673dd4441d697e342b3996e85134d2a4d151ccef4bd60bed3f: Status 404 returned error can't find the container with id b66fd0f5ef521d673dd4441d697e342b3996e85134d2a4d151ccef4bd60bed3f Oct 30 03:09:46.173151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:46.173123 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" event={"ID":"504a6d3e-f83e-4c3e-92a7-0db908fcfe00","Type":"ContainerStarted","Data":"b66fd0f5ef521d673dd4441d697e342b3996e85134d2a4d151ccef4bd60bed3f"} Oct 30 03:09:47.178756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:47.178718 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" event={"ID":"504a6d3e-f83e-4c3e-92a7-0db908fcfe00","Type":"ContainerStarted","Data":"10eb983ac17e6dc431281d6a3a8ff263409b8e923e7d64ec0837310511904a42"} Oct 30 03:09:47.179150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:47.178785 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:09:47.222740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:09:47.222684 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" podStartSLOduration=1.848937448 podStartE2EDuration="2.222668136s" totalImagesPullingTime="373.730688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:47.221275529 +0000 UTC m=+415.206395421" watchObservedRunningTime="2026-10-30 03:09:47.222668136 +0000 UTC m=+415.207788024" Oct 30 03:10:11.749180 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:10:11.749139 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:10:18.184571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.184539 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jk86r" Oct 30 03:10:18.251758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.251719 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:10:18.252051 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.252010 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" podUID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" containerName="manager" containerID="cri-o://f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa" gracePeriod=30 Oct 30 03:10:18.494178 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.494154 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:10:18.617240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.617199 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\" (UniqueName: \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\") pod \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " Oct 30 03:10:18.617406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.617254 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\" (UniqueName: \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\") pod \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\" (UID: \"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c\") " Oct 30 03:10:18.619399 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.619370 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m" pod "c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" (UID: "c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c"). InnerVolumeSpecName "kube-api-access-62v7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:10:18.619479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.619423 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert" pod "c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" (UID: "c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:10:18.718269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.718238 4560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:10:18.718269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:18.718264 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62v7m\" (UniqueName: \"kubernetes.io/projected/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c-kube-api-access-62v7m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:10:19.321723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.321685 4560 generic.go:356] "Generic (PLEG): container finished" podID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" containerID="f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa" exitCode=0 Oct 30 03:10:19.322186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.321734 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" Oct 30 03:10:19.322186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.321765 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" event={"ID":"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c","Type":"ContainerDied","Data":"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa"} Oct 30 03:10:19.322186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.321812 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp" event={"ID":"c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c","Type":"ContainerDied","Data":"4101714bfcc86df5483b023d64247d20e4d89bd0d1af2ea3de27fc05e7d37f78"} Oct 30 03:10:19.322186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.321839 4560 scope.go:122] "RemoveContainer" containerID="f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa" Oct 30 03:10:19.330621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.330605 4560 scope.go:122] "RemoveContainer" containerID="f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa" Oct 30 03:10:19.330881 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:10:19.330862 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa\": container with ID starting with f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa not found: ID does not exist" containerID="f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa" Oct 30 03:10:19.330987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.330890 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa"} err="failed to get container status \"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa\": rpc error: code = NotFound desc = could not find container \"f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa\": container with ID starting with f8947c3c694c552a1bdbc694a6d9d8f25fb6c127b2c1d117fcaa800a69e6befa not found: ID does not exist" Oct 30 03:10:19.374915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.374883 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:10:19.387545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:19.387501 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lp4cp"] Oct 30 03:10:20.151494 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:10:20.151456 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" path="/var/lib/kubelet/pods/c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c/volumes" Oct 30 03:10:41.748854 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:10:41.748814 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:11:11.751337 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:11:11.751248 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:11:17.341188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.341151 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:11:17.341744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.341724 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" containerName="manager" Oct 30 03:11:17.341823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.341749 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" containerName="manager" Oct 30 03:11:17.341877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.341851 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75b4ca1-ffa0-41c2-8244-d94da2ed3d7c" containerName="manager" Oct 30 03:11:17.345210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.345190 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359652 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359712 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359777 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd22\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359849 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359878 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.359946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359932 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.360108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.359958 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.360108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.360020 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.360108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.360042 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.363202 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.363181 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 03:11:17.377588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.377558 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:11:17.431733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.431696 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fqfn6\"" Oct 30 03:11:17.460930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.460891 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.460930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.460935 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.460961 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd22\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.460988 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461013 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461031 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461052 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461454 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461478 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461649 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461468 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461649 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461550 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461840 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461790 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.461987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.461968 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.462267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.462240 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.463445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.463423 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.463998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.463977 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.480812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.480784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.480907 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.480893 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd22\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpfhc\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.660260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.660172 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:17.981003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:17.980976 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:11:17.983189 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:11:17.983157 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1491d054_52a8_4690_b522_dd13c03fab34.slice/crio-6cd11055d42e76411da86699367dbb4ee12f2e6a9b621ecb561601c6cc4252d5 WatchSource:0}: Error finding container 6cd11055d42e76411da86699367dbb4ee12f2e6a9b621ecb561601c6cc4252d5: Status 404 returned error can't find the container with id 6cd11055d42e76411da86699367dbb4ee12f2e6a9b621ecb561601c6cc4252d5 Oct 30 03:11:18.592268 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:18.592231 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" event={"ID":"1491d054-52a8-4690-b522-dd13c03fab34","Type":"ContainerStarted","Data":"6cd11055d42e76411da86699367dbb4ee12f2e6a9b621ecb561601c6cc4252d5"} Oct 30 03:11:20.861500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:20.861469 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:11:20.861774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:20.861563 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:11:20.861774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:20.861595 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:11:21.608614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:21.608576 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" event={"ID":"1491d054-52a8-4690-b522-dd13c03fab34","Type":"ContainerStarted","Data":"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b"} Oct 30 03:11:21.674675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:21.670855 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:21.674675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:21.674578 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:21.674675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:21.674630 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:22.662140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:22.662103 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:22.662756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:22.662726 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:23.669540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:23.663870 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:23.669540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:23.663929 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:24.667540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:24.663888 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:24.667540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:24.663949 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:25.668539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:25.666879 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:25.668539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:25.666950 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:26.666927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:26.666890 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:26.667204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:26.667176 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:27.660983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:27.660948 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 03:11:27.661394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:27.661003 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 03:11:29.118624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:29.118588 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:29.119591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:29.119561 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:29.120369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:29.120354 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:11:29.182208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:11:29.182153 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podStartSLOduration=9.306724738 podStartE2EDuration="12.182139347s" totalImagesPullingTime="2.875414609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:21.674645901 +0000 UTC m=+509.659765800" watchObservedRunningTime="2026-10-30 03:11:29.182139347 +0000 UTC m=+517.167259308" Oct 30 03:11:41.748893 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:11:41.748857 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:12:11.748794 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:12:11.748694 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:12:17.208659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.208626 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb"] Oct 30 03:12:17.212329 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.212312 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.227661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.227637 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:12:17.240106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.240073 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:12:17.240106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.240084 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:12:17.242152 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.242129 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb"] Oct 30 03:12:17.328298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.328270 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:12:17.333764 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.333727 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.333920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.333780 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.333920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.333905 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn44\" (UniqueName: \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.435078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.435036 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.435078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.435081 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.435290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.435117 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn44\" (UniqueName: \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.435966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.435940 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.437466 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.437439 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.463621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.463555 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn44\" (UniqueName: \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\") pod \"maas-api-revoke-keys-e2e-ait-421ef828-p6nxb\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.525541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.524682 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:17.863918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:17.863892 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb"] Oct 30 03:12:17.866061 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:12:17.866034 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cd4a63_5dfa_4a20_bbfc_fffa471a0aec.slice/crio-2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182 WatchSource:0}: Error finding container 2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182: Status 404 returned error can't find the container with id 2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182 Oct 30 03:12:18.875510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:18.875468 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" event={"ID":"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec","Type":"ContainerStarted","Data":"2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182"} Oct 30 03:12:20.886034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:20.886002 4560 generic.go:356] "Generic (PLEG): container finished" podID="98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" containerID="3ac929e3059d8baa15a1c0c19ec5373d4fc638d9d6b26f4225a5d0ce206f0d98" exitCode=0 Oct 30 03:12:20.886395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:20.886079 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" event={"ID":"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec","Type":"ContainerDied","Data":"3ac929e3059d8baa15a1c0c19ec5373d4fc638d9d6b26f4225a5d0ce206f0d98"} Oct 30 03:12:22.028653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.028630 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:22.205073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.205041 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\" (UniqueName: \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\") pod \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " Oct 30 03:12:22.205256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.205084 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\") pod \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " Oct 30 03:12:22.205256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.205142 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\") pod \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\" (UID: \"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec\") " Oct 30 03:12:22.205492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.205465 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca" pod "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" (UID: "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:22.207420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.207390 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca" pod "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" (UID: "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:22.207527 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.207473 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44" pod "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" (UID: "98cd4a63-5dfa-4a20-bbfc-fffa471a0aec"). InnerVolumeSpecName "kube-api-access-jzn44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:22.306837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.306803 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzn44\" (UniqueName: \"kubernetes.io/projected/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-kube-api-access-jzn44\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:22.306837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.306832 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:22.306837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.306842 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:22.897358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.897320 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" event={"ID":"98cd4a63-5dfa-4a20-bbfc-fffa471a0aec","Type":"ContainerDied","Data":"2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182"} Oct 30 03:12:22.897358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.897354 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb" Oct 30 03:12:22.897604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:22.897374 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2169c2e73ee4e8485413f98676103429361d9255d9313c71dd8c7edeb3fc0182" Oct 30 03:12:26.406795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.406764 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4"] Oct 30 03:12:26.407174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.407150 4560 cpu_manager.go:414] "containerMap: removing container" podUID="98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" containerName="revoke-keys" Oct 30 03:12:26.407174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.407161 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" containerName="revoke-keys" Oct 30 03:12:26.407249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.407229 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" containerName="revoke-keys" Oct 30 03:12:26.410397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.410380 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.426947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.426925 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:12:26.432282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.432250 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4"] Oct 30 03:12:26.440684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.440662 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.440798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.440770 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.440858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.440796 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254m5\" (UniqueName: \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.441304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.441288 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:12:26.441304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.441293 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:12:26.526662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.526636 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:12:26.543699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.542246 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.543699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.542310 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254m5\" (UniqueName: \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.543699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.542391 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.543699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.543270 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.548406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.546396 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.572255 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.572223 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254m5\" (UniqueName: \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\") pod \"maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:26.724310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:26.724276 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:27.037566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:27.037476 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4"] Oct 30 03:12:27.041395 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:12:27.041362 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f328a6_1e2a_4cd3_aaff_74db18f4244d.slice/crio-b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56 WatchSource:0}: Error finding container b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56: Status 404 returned error can't find the container with id b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56 Oct 30 03:12:27.923396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:27.923354 4560 generic.go:356] "Generic (PLEG): container finished" podID="58f328a6-1e2a-4cd3-aaff-74db18f4244d" containerID="70c132910d83e1c24b25b7dc8fdc07b96013f186036ea66bd07cabe2c5d284e6" exitCode=0 Oct 30 03:12:27.923786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:27.923428 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" event={"ID":"58f328a6-1e2a-4cd3-aaff-74db18f4244d","Type":"ContainerDied","Data":"70c132910d83e1c24b25b7dc8fdc07b96013f186036ea66bd07cabe2c5d284e6"} Oct 30 03:12:27.923786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:27.923481 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" event={"ID":"58f328a6-1e2a-4cd3-aaff-74db18f4244d","Type":"ContainerStarted","Data":"b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56"} Oct 30 03:12:29.059213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.059185 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:29.065258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.065235 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\") pod \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " Oct 30 03:12:29.065332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.065276 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\") pod \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " Oct 30 03:12:29.065332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.065320 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\" (UniqueName: \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\") pod \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\" (UID: \"58f328a6-1e2a-4cd3-aaff-74db18f4244d\") " Oct 30 03:12:29.065668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.065644 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca" pod "58f328a6-1e2a-4cd3-aaff-74db18f4244d" (UID: "58f328a6-1e2a-4cd3-aaff-74db18f4244d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:29.067156 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.067131 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca" pod "58f328a6-1e2a-4cd3-aaff-74db18f4244d" (UID: "58f328a6-1e2a-4cd3-aaff-74db18f4244d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:29.067311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.067294 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5" pod "58f328a6-1e2a-4cd3-aaff-74db18f4244d" (UID: "58f328a6-1e2a-4cd3-aaff-74db18f4244d"). InnerVolumeSpecName "kube-api-access-254m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:29.167034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.167002 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:29.167034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.167036 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f328a6-1e2a-4cd3-aaff-74db18f4244d-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:29.167247 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.167054 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254m5\" (UniqueName: \"kubernetes.io/projected/58f328a6-1e2a-4cd3-aaff-74db18f4244d-kube-api-access-254m5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:12:29.933612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.933583 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" Oct 30 03:12:29.933785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.933582 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4" event={"ID":"58f328a6-1e2a-4cd3-aaff-74db18f4244d","Type":"ContainerDied","Data":"b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56"} Oct 30 03:12:29.933785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:29.933705 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e8ce0bf7183b97c9aa822fefc26d21e60ab1f795ce1a7362c42fa4b083bd56" Oct 30 03:12:38.168648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:38.168556 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb"] Oct 30 03:12:38.181689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:38.181658 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421ef828-p6nxb"] Oct 30 03:12:40.152934 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:40.152894 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cd4a63-5dfa-4a20-bbfc-fffa471a0aec" path="/var/lib/kubelet/pods/98cd4a63-5dfa-4a20-bbfc-fffa471a0aec/volumes" Oct 30 03:12:40.555228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:40.555195 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4"] Oct 30 03:12:40.569211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:40.569176 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c7c56c3d-7jnn4"] Oct 30 03:12:41.750290 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:12:41.750255 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:12:42.150837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:12:42.150760 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f328a6-1e2a-4cd3-aaff-74db18f4244d" path="/var/lib/kubelet/pods/58f328a6-1e2a-4cd3-aaff-74db18f4244d/volumes" Oct 30 03:13:01.984164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:01.984126 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 03:13:01.984602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:01.984575 4560 cpu_manager.go:414] "containerMap: removing container" podUID="58f328a6-1e2a-4cd3-aaff-74db18f4244d" containerName="revoke-keys" Oct 30 03:13:01.984602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:01.984589 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f328a6-1e2a-4cd3-aaff-74db18f4244d" containerName="revoke-keys" Oct 30 03:13:01.984713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:01.984700 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f328a6-1e2a-4cd3-aaff-74db18f4244d" containerName="revoke-keys" Oct 30 03:13:01.992477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:01.992452 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.010340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.010314 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 03:13:02.021647 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.021623 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"kube-root-ca.crt\"" Oct 30 03:13:02.021802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.021657 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"openshift-service-ca.crt\"" Oct 30 03:13:02.021802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.021727 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"default-dockercfg-48q8f\"" Oct 30 03:13:02.021802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.021772 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"simulator-w0-a9c9b6-kserve-self-signed-certs\"" Oct 30 03:13:02.078196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078165 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.078388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078209 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mhr\" (UniqueName: \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.078388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078310 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.078506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078392 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.078506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078425 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.078637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.078565 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179405 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179463 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179493 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179565 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179602 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179641 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mhr\" (UniqueName: \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.179933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.180006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.179983 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.180060 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.180021 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.181699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.181676 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.181971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.181956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.202496 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.202467 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mhr\" (UniqueName: \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\") pod \"simulator-w0-a9c9b6-kserve-58675477f7-8ndd5\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.306168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.306076 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:02.607629 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:02.607600 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 03:13:02.609043 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:13:02.609011 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e9d1e24_9e81_47d0_81fc_85f9662c41bc.slice/crio-e5300195045ce111a193acd94cb2dbbf0eadf2864fcdead32c05843553c803a8 WatchSource:0}: Error finding container e5300195045ce111a193acd94cb2dbbf0eadf2864fcdead32c05843553c803a8: Status 404 returned error can't find the container with id e5300195045ce111a193acd94cb2dbbf0eadf2864fcdead32c05843553c803a8 Oct 30 03:13:03.092568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:03.092530 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" event={"ID":"4e9d1e24-9e81-47d0-81fc-85f9662c41bc","Type":"ContainerStarted","Data":"e5300195045ce111a193acd94cb2dbbf0eadf2864fcdead32c05843553c803a8"} Oct 30 03:13:04.098818 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:04.098780 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" event={"ID":"4e9d1e24-9e81-47d0-81fc-85f9662c41bc","Type":"ContainerStarted","Data":"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e"} Oct 30 03:13:04.140871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:04.140816 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" podStartSLOduration=2.081909218 podStartE2EDuration="3.140802763s" totalImagesPullingTime="1.058893545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:04.139754951 +0000 UTC m=+612.124874840" watchObservedRunningTime="2026-10-30 03:13:04.140802763 +0000 UTC m=+612.125922652" Oct 30 03:13:04.306314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:04.306284 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:10.796341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.796305 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 03:13:10.800729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.800702 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.823766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.823718 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"openshift-service-ca.crt\"" Oct 30 03:13:10.830374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.830353 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"kube-root-ca.crt\"" Oct 30 03:13:10.836497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.836477 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"simulator-w2-2d5b09-kserve-self-signed-certs\"" Oct 30 03:13:10.836497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.836491 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"default-dockercfg-rlsgn\"" Oct 30 03:13:10.838402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.838381 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 03:13:10.861728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.861696 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.861728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.861743 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.861967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.861822 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.861967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.861939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.862073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.861979 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.862073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.862007 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962445 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962484 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962504 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962611 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962643 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962696 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.962990 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.962931 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.963049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.963003 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.963182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.963162 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.965372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.965343 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.965751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.965730 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:10.982896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:10.982871 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\") pod \"simulator-w2-2d5b09-kserve-7689b5f567-krmcd\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:11.114827 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:11.114739 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:11.417726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:11.417699 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 03:13:11.419141 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:13:11.419110 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7a218a_b2e1_4687_8f6c_ae21f916004f.slice/crio-8b4fa1c9978db6d303d37379f36dde3b8a03d6ebdea576afb97dab6bc4c71b88 WatchSource:0}: Error finding container 8b4fa1c9978db6d303d37379f36dde3b8a03d6ebdea576afb97dab6bc4c71b88: Status 404 returned error can't find the container with id 8b4fa1c9978db6d303d37379f36dde3b8a03d6ebdea576afb97dab6bc4c71b88 Oct 30 03:13:11.749894 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:13:11.749855 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:13:12.138177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:12.138095 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" event={"ID":"ba7a218a-b2e1-4687-8f6c-ae21f916004f","Type":"ContainerStarted","Data":"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca"} Oct 30 03:13:12.138177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:12.138139 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" event={"ID":"ba7a218a-b2e1-4687-8f6c-ae21f916004f","Type":"ContainerStarted","Data":"8b4fa1c9978db6d303d37379f36dde3b8a03d6ebdea576afb97dab6bc4c71b88"} Oct 30 03:13:12.181028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:12.180973 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" podStartSLOduration=2.180958264 podStartE2EDuration="2.180958264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:12.179377468 +0000 UTC m=+620.164497356" watchObservedRunningTime="2026-10-30 03:13:12.180958264 +0000 UTC m=+620.166078152" Oct 30 03:13:12.306659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:12.306626 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:12.322454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:12.322428 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:13.115500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:13.115470 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:13.154440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:13.154404 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 03:13:20.386593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.386554 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 03:13:20.389769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.389743 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.425894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.425860 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"openshift-service-ca.crt\"" Oct 30 03:13:20.426082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.425932 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"kube-root-ca.crt\"" Oct 30 03:13:20.426082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.426036 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"default-dockercfg-czx8v\"" Oct 30 03:13:20.427776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.427752 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 03:13:20.427994 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.427978 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"simulator-w4-8d80b1-kserve-self-signed-certs\"" Oct 30 03:13:20.450898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.450861 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.451058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.450929 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.451058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.451021 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl89s\" (UniqueName: \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.451151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.451077 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.451212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.451163 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.451271 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.451250 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.552973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.552937 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.552987 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl89s\" (UniqueName: \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553079 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553135 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553185 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553444 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.553744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.553719 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.556071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.556042 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.556186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.556089 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.587006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.586977 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl89s\" (UniqueName: \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\") pod \"simulator-w4-8d80b1-kserve-576844c4bd-cdsl6\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:20.703596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:20.703556 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:21.009009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.008923 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 03:13:21.017320 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:13:21.017287 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee2fe47_8e05_4edc_b0c2_b15bdac78880.slice/crio-3aab08d2eff59e8752146becb2f3ee4bfca32843237c31c1f734942d3a05c26a WatchSource:0}: Error finding container 3aab08d2eff59e8752146becb2f3ee4bfca32843237c31c1f734942d3a05c26a: Status 404 returned error can't find the container with id 3aab08d2eff59e8752146becb2f3ee4bfca32843237c31c1f734942d3a05c26a Oct 30 03:13:21.115857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.115826 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:21.128549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.128504 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:21.185044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.184996 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" event={"ID":"eee2fe47-8e05-4edc-b0c2-b15bdac78880","Type":"ContainerStarted","Data":"b901c16564919170acfc54051adb0913066ef802acbf42ef21ee1818b1551ddc"} Oct 30 03:13:21.185237 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.185057 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" event={"ID":"eee2fe47-8e05-4edc-b0c2-b15bdac78880","Type":"ContainerStarted","Data":"3aab08d2eff59e8752146becb2f3ee4bfca32843237c31c1f734942d3a05c26a"} Oct 30 03:13:21.197068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.197037 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 03:13:21.235782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.235731 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" podStartSLOduration=1.235717377 podStartE2EDuration="1.235717377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:21.234366716 +0000 UTC m=+629.219486617" watchObservedRunningTime="2026-10-30 03:13:21.235717377 +0000 UTC m=+629.220837265" Oct 30 03:13:21.704626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:21.704594 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:30.704013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:30.703968 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:30.716598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:30.716570 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:31.244777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.244748 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 03:13:31.248453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.248427 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.251726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.251694 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 03:13:31.264737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.264712 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"premium-w0-a9c9b6-kserve-self-signed-certs\"" Oct 30 03:13:31.279350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.279321 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 03:13:31.361699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361657 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.361885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361744 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.361885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361777 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.361885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361827 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.362054 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361925 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.362054 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.361959 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfm94\" (UniqueName: \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463187 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463231 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfm94\" (UniqueName: \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463253 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463326 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463359 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463395 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.463881 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463845 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.464005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463924 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.464005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.463955 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.465796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.465772 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.465912 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.465880 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.484377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.484344 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfm94\" (UniqueName: \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\") pod \"premium-w0-a9c9b6-kserve-5cf68468d8-r58n5\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.563926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.563849 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:31.868055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:31.868023 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 03:13:32.243017 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:32.242980 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" event={"ID":"e556d36d-074d-4f34-b96d-5c6b9b35dd77","Type":"ContainerStarted","Data":"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1"} Oct 30 03:13:32.243223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:32.243029 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" event={"ID":"e556d36d-074d-4f34-b96d-5c6b9b35dd77","Type":"ContainerStarted","Data":"81a981d8dfbc85d9d5d8d31681b80ee5da20cd076b17e6f74559029020860825"} Oct 30 03:13:32.564578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:32.564485 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:38.096305 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.096253 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" podStartSLOduration=7.096235993 podStartE2EDuration="7.096235993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:32.286675638 +0000 UTC m=+640.271795528" watchObservedRunningTime="2026-10-30 03:13:38.096235993 +0000 UTC m=+646.081355880" Oct 30 03:13:38.098009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.097981 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 03:13:38.101831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.101806 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.125065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.125036 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 03:13:38.135321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.135294 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"premium-w2-2d5b09-kserve-self-signed-certs\"" Oct 30 03:13:38.234182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234148 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.234386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234198 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.234386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234319 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5jv\" (UniqueName: \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.234386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234346 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.234386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234363 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.234663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.234480 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335075 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5jv\" (UniqueName: \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335116 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335141 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335174 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335260 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335616 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335730 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.335760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.335754 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.337653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.337634 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.337871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.337854 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.355498 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.355437 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5jv\" (UniqueName: \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\") pod \"premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.415025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.414992 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:38.719456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:38.719430 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 03:13:38.721124 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:13:38.721096 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66840185_4102_4042_a065_d3a251a039d2.slice/crio-81b2450f9226f1c16f3cf4f84f1f04945c39397b20fa8e2d7ce7b484029ef1f7 WatchSource:0}: Error finding container 81b2450f9226f1c16f3cf4f84f1f04945c39397b20fa8e2d7ce7b484029ef1f7: Status 404 returned error can't find the container with id 81b2450f9226f1c16f3cf4f84f1f04945c39397b20fa8e2d7ce7b484029ef1f7 Oct 30 03:13:39.279104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:39.279065 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" event={"ID":"66840185-4102-4042-a065-d3a251a039d2","Type":"ContainerStarted","Data":"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13"} Oct 30 03:13:39.279541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:39.279113 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" event={"ID":"66840185-4102-4042-a065-d3a251a039d2","Type":"ContainerStarted","Data":"81b2450f9226f1c16f3cf4f84f1f04945c39397b20fa8e2d7ce7b484029ef1f7"} Oct 30 03:13:39.322320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:39.322272 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" podStartSLOduration=1.3222591160000001 podStartE2EDuration="1.322259116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:39.321698429 +0000 UTC m=+647.306818318" watchObservedRunningTime="2026-10-30 03:13:39.322259116 +0000 UTC m=+647.307379004" Oct 30 03:13:39.416102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:39.416070 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:41.563996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:41.563959 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:41.576555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:41.576511 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:41.749091 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:13:41.749061 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:13:42.305899 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:42.305871 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 03:13:48.415758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:48.415724 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:48.428890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:48.428859 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:13:49.342193 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:13:49.342168 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 03:14:11.748647 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:14:11.748565 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:14:33.657471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.657434 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd"] Oct 30 03:14:33.661047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.661029 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.676270 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.676243 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.676397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.676286 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.676397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.676303 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.689820 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.689787 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:14:33.689897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.689885 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:14:33.689937 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.689908 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:14:33.689969 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.689951 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:14:33.707692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.707665 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd"] Oct 30 03:14:33.777815 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.777774 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.778019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.777836 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.778019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.777863 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.778568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.778548 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.780120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.780097 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.798691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.798661 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\") pod \"maas-api-revoke-sub-09e70622-hjsjd\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:33.976180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:33.976152 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:34.289342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:34.289316 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd"] Oct 30 03:14:34.290369 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:14:34.290339 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f02e51b_d369_4460_a9a0_59eef0fc9282.slice/crio-d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c WatchSource:0}: Error finding container d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c: Status 404 returned error can't find the container with id d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c Oct 30 03:14:34.572246 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:34.572215 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" event={"ID":"0f02e51b-d369-4460-a9a0-59eef0fc9282","Type":"ContainerStarted","Data":"f44ddaf0fdebe64eb73e8f25c25636b4a542d09d95e87a2145cea5d745d5693b"} Oct 30 03:14:34.572406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:34.572271 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" event={"ID":"0f02e51b-d369-4460-a9a0-59eef0fc9282","Type":"ContainerStarted","Data":"d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c"} Oct 30 03:14:34.618757 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:34.618699 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" podStartSLOduration=1.618685084 podStartE2EDuration="1.618685084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:34.616973985 +0000 UTC m=+702.602093873" watchObservedRunningTime="2026-10-30 03:14:34.618685084 +0000 UTC m=+702.603804992" Oct 30 03:14:35.579028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:35.578995 4560 generic.go:356] "Generic (PLEG): container finished" podID="0f02e51b-d369-4460-a9a0-59eef0fc9282" containerID="f44ddaf0fdebe64eb73e8f25c25636b4a542d09d95e87a2145cea5d745d5693b" exitCode=0 Oct 30 03:14:35.579445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:35.579086 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" event={"ID":"0f02e51b-d369-4460-a9a0-59eef0fc9282","Type":"ContainerDied","Data":"f44ddaf0fdebe64eb73e8f25c25636b4a542d09d95e87a2145cea5d745d5693b"} Oct 30 03:14:36.709437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.709408 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:36.805012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.804982 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\") pod \"0f02e51b-d369-4460-a9a0-59eef0fc9282\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " Oct 30 03:14:36.805174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.805081 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\") pod \"0f02e51b-d369-4460-a9a0-59eef0fc9282\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " Oct 30 03:14:36.805174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.805140 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\") pod \"0f02e51b-d369-4460-a9a0-59eef0fc9282\" (UID: \"0f02e51b-d369-4460-a9a0-59eef0fc9282\") " Oct 30 03:14:36.805537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.805493 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca" pod "0f02e51b-d369-4460-a9a0-59eef0fc9282" (UID: "0f02e51b-d369-4460-a9a0-59eef0fc9282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:14:36.807030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.807009 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn" pod "0f02e51b-d369-4460-a9a0-59eef0fc9282" (UID: "0f02e51b-d369-4460-a9a0-59eef0fc9282"). InnerVolumeSpecName "kube-api-access-phwkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:14:36.807138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.807043 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca" pod "0f02e51b-d369-4460-a9a0-59eef0fc9282" (UID: "0f02e51b-d369-4460-a9a0-59eef0fc9282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:14:36.906113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.906033 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:14:36.906113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.906064 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f02e51b-d369-4460-a9a0-59eef0fc9282-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:14:36.906113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:36.906075 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/0f02e51b-d369-4460-a9a0-59eef0fc9282-kube-api-access-phwkn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:14:37.591413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:37.591377 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" Oct 30 03:14:37.591610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:37.591412 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd" event={"ID":"0f02e51b-d369-4460-a9a0-59eef0fc9282","Type":"ContainerDied","Data":"d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c"} Oct 30 03:14:37.591610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:37.591461 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e06d94ee8f62c25c0265307b9ac48920d36703bf87051c7919f2f169d6483c" Oct 30 03:14:40.545818 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.545783 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:14:40.546312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.546259 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0f02e51b-d369-4460-a9a0-59eef0fc9282" containerName="revoke-keys" Oct 30 03:14:40.546312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.546274 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f02e51b-d369-4460-a9a0-59eef0fc9282" containerName="revoke-keys" Oct 30 03:14:40.546886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.546431 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f02e51b-d369-4460-a9a0-59eef0fc9282" containerName="revoke-keys" Oct 30 03:14:40.549909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.549891 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.565551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.565509 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a9c9b6-openshift-default-dockercfg-9jv62\"" Oct 30 03:14:40.581090 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.581058 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:14:40.640716 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640681 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.640716 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640716 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.640916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640852 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.640916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640884 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.640984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640920 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.641016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.640991 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7kw\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.641049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.641030 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.641104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.641091 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.641140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.641113 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742185 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742224 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742271 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742302 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7kw\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742330 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742366 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742423 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742446 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742716 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742798 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742899 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.742965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.742923 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.743198 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.743176 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.744793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.744771 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.744973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.744953 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.763571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.763541 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.764246 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.764228 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7kw\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:40.863038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:40.862947 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:41.177840 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.177814 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:14:41.178391 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:14:41.178369 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671fed3c_4bdf_442e_968f_030678e14961.slice/crio-bfd02a4973a911178b1d8fb31178d9d8e591556c9bbb1dba200fae95bc34166b WatchSource:0}: Error finding container bfd02a4973a911178b1d8fb31178d9d8e591556c9bbb1dba200fae95bc34166b: Status 404 returned error can't find the container with id bfd02a4973a911178b1d8fb31178d9d8e591556c9bbb1dba200fae95bc34166b Oct 30 03:14:41.180620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.180593 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:14:41.180691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.180654 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:14:41.180691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.180680 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:14:41.613951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.613916 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" event={"ID":"671fed3c-4bdf-442e-968f-030678e14961","Type":"ContainerStarted","Data":"ee07daeb9627703104bb9b9d75886adff480b48a6fc86b95a0d1014f7f195d24"} Oct 30 03:14:41.614400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.613971 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" event={"ID":"671fed3c-4bdf-442e-968f-030678e14961","Type":"ContainerStarted","Data":"bfd02a4973a911178b1d8fb31178d9d8e591556c9bbb1dba200fae95bc34166b"} Oct 30 03:14:41.657365 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.657080 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podStartSLOduration=1.657065282 podStartE2EDuration="1.657065282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:41.656972402 +0000 UTC m=+709.642092302" watchObservedRunningTime="2026-10-30 03:14:41.657065282 +0000 UTC m=+709.642185232" Oct 30 03:14:41.748838 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:14:41.748795 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:14:41.863135 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:41.863105 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:42.873413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:42.873364 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:42.874081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:42.874033 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:43.867358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:43.866900 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:43.867358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:43.866962 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:44.872605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:44.871875 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:44.872605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:44.871944 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:45.869945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:45.869908 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:45.870200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:45.870170 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:46.873689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:46.873651 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:46.874294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:46.874261 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:47.863634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:47.863596 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:14:47.863802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:47.863654 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 03:14:48.903322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:48.903287 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:48.903860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:48.903674 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:48.904737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:48.904716 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:14:59.101022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.100988 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:14:59.104745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.104723 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.135197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.135146 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"openshift-service-ca.crt\"" Oct 30 03:14:59.135350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.135303 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"kube-root-ca.crt\"" Oct 30 03:14:59.135399 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.135383 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"default-dockercfg-cqsbs\"" Oct 30 03:14:59.135445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.135389 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"e2e-conflict-a-9efa19e9-kserve-self-signed-certs\"" Oct 30 03:14:59.138656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.138627 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:14:59.245144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245111 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.245317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245229 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.245317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245270 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76b7f\" (UniqueName: \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.245429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245334 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.245429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245417 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.245496 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.245443 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.346922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.346883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.346935 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76b7f\" (UniqueName: \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.346997 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347095 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347285 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347137 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347285 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347198 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347304 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347419 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.347641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.347614 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.349388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.349363 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.349572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.349556 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.367321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.367255 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76b7f\" (UniqueName: \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\") pod \"e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.419217 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.419183 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:14:59.722616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:14:59.722538 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:14:59.724412 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:14:59.724386 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod859adab4_61bf_4461_b91f_927ceb60eb4f.slice/crio-dcc1ca5bb19507209d906507aadcfe81ad66964f436b86958f816eef4608ad66 WatchSource:0}: Error finding container dcc1ca5bb19507209d906507aadcfe81ad66964f436b86958f816eef4608ad66: Status 404 returned error can't find the container with id dcc1ca5bb19507209d906507aadcfe81ad66964f436b86958f816eef4608ad66 Oct 30 03:15:00.153081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.153003 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:00.156747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.156725 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:00.156872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.156843 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.160276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.160253 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.164269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.164245 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:00.168622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.168600 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.171167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.171143 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:00.179858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.178530 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:00.181819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.181700 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:15:00.181932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.181841 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:15:00.190340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.190309 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:00.257322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257282 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.257491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257336 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6cf9\" (UniqueName: \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.257491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257473 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.257624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257504 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75px\" (UniqueName: \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.257624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257544 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpd74\" (UniqueName: \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.257715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.257632 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.275264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.275233 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:15:00.359196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359167 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.359376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359257 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.359376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359287 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6cf9\" (UniqueName: \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.359544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359398 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.359544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359436 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75px\" (UniqueName: \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.359544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359464 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpd74\" (UniqueName: \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.360008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.360095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.359985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.360095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.360073 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.382010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.381982 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6cf9\" (UniqueName: \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.382203 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.382181 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpd74\" (UniqueName: \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.382245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.382193 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75px\" (UniqueName: \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.472697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.472667 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:00.489411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.489386 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:00.502288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.502257 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:00.719231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.719190 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" event={"ID":"859adab4-61bf-4461-b91f-927ceb60eb4f","Type":"ContainerStarted","Data":"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae"} Oct 30 03:15:00.719411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.719247 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" event={"ID":"859adab4-61bf-4461-b91f-927ceb60eb4f","Type":"ContainerStarted","Data":"dcc1ca5bb19507209d906507aadcfe81ad66964f436b86958f816eef4608ad66"} Oct 30 03:15:00.764071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.763981 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" podStartSLOduration=1.763966338 podStartE2EDuration="1.763966338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:00.763212237 +0000 UTC m=+728.748332125" watchObservedRunningTime="2026-10-30 03:15:00.763966338 +0000 UTC m=+728.749086225" Oct 30 03:15:00.784629 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.784597 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:00.788982 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:00.788954 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a9d734_958f_471e_a36c_3a953e288997.slice/crio-f8bcfc414da4dedfe9ae50b9dce0a77dfe4f097618434daa1865f301fbcc5891 WatchSource:0}: Error finding container f8bcfc414da4dedfe9ae50b9dce0a77dfe4f097618434daa1865f301fbcc5891: Status 404 returned error can't find the container with id f8bcfc414da4dedfe9ae50b9dce0a77dfe4f097618434daa1865f301fbcc5891 Oct 30 03:15:00.804169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.804146 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:00.804821 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:00.804797 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ff2498_3b3b_40d2_a3f8_a076c4cf11d8.slice/crio-c4c9a90acd94e4b962d0efbc5e7adddb1f3d9b341dde88744e4ab380665e922c WatchSource:0}: Error finding container c4c9a90acd94e4b962d0efbc5e7adddb1f3d9b341dde88744e4ab380665e922c: Status 404 returned error can't find the container with id c4c9a90acd94e4b962d0efbc5e7adddb1f3d9b341dde88744e4ab380665e922c Oct 30 03:15:00.834116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:00.834092 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:00.834513 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:00.834491 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb269495f_a153_4dd7_9a7b_bc2d702cc929.slice/crio-d4be5da845000d90327935777c3e173834e6914c428443fc9896784cf281d206 WatchSource:0}: Error finding container d4be5da845000d90327935777c3e173834e6914c428443fc9896784cf281d206: Status 404 returned error can't find the container with id d4be5da845000d90327935777c3e173834e6914c428443fc9896784cf281d206 Oct 30 03:15:01.419544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.419451 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:15:01.726949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.726919 4560 generic.go:356] "Generic (PLEG): container finished" podID="36a9d734-958f-471e-a36c-3a953e288997" containerID="4dd5a5d156692149c29bad06eec8653865f7d5546e067dca0e13383638ec7ac4" exitCode=60 Oct 30 03:15:01.727072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.726966 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerDied","Data":"4dd5a5d156692149c29bad06eec8653865f7d5546e067dca0e13383638ec7ac4"} Oct 30 03:15:01.727072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.727020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerStarted","Data":"f8bcfc414da4dedfe9ae50b9dce0a77dfe4f097618434daa1865f301fbcc5891"} Oct 30 03:15:01.727300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.727280 4560 scope.go:122] "RemoveContainer" containerID="4dd5a5d156692149c29bad06eec8653865f7d5546e067dca0e13383638ec7ac4" Oct 30 03:15:01.728705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.728681 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerStarted","Data":"9930de1860ed81baca1284740a91f1a926cabad3190cfd293f4f71662b2a4566"} Oct 30 03:15:01.728810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.728733 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerStarted","Data":"d4be5da845000d90327935777c3e173834e6914c428443fc9896784cf281d206"} Oct 30 03:15:01.730348 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.730243 4560 generic.go:356] "Generic (PLEG): container finished" podID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerID="5f5a8d73de4e35f86fbdd92396b018f1e6766f20774e35def55a569f9d8d8516" exitCode=60 Oct 30 03:15:01.730348 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.730316 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerDied","Data":"5f5a8d73de4e35f86fbdd92396b018f1e6766f20774e35def55a569f9d8d8516"} Oct 30 03:15:01.730543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.730356 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerStarted","Data":"c4c9a90acd94e4b962d0efbc5e7adddb1f3d9b341dde88744e4ab380665e922c"} Oct 30 03:15:01.730543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.730475 4560 scope.go:122] "RemoveContainer" containerID="5f5a8d73de4e35f86fbdd92396b018f1e6766f20774e35def55a569f9d8d8516" Oct 30 03:15:01.872083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:01.872037 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" podStartSLOduration=1.872024583 podStartE2EDuration="1.872024583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:01.871135867 +0000 UTC m=+729.856255766" watchObservedRunningTime="2026-10-30 03:15:01.872024583 +0000 UTC m=+729.857144471" Oct 30 03:15:02.737941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.737912 4560 generic.go:356] "Generic (PLEG): container finished" podID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerID="8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255" exitCode=60 Oct 30 03:15:02.738349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.737997 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerDied","Data":"8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255"} Oct 30 03:15:02.738349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.738065 4560 scope.go:122] "RemoveContainer" containerID="5f5a8d73de4e35f86fbdd92396b018f1e6766f20774e35def55a569f9d8d8516" Oct 30 03:15:02.738349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.738253 4560 scope.go:122] "RemoveContainer" containerID="8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255" Oct 30 03:15:02.738567 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:02.738545 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s_odh-ai-gateway-infra(60ff2498-3b3b-40d2-a3f8-a076c4cf11d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" Oct 30 03:15:02.740126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.740106 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerStarted","Data":"04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce"} Oct 30 03:15:02.741474 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.741452 4560 generic.go:356] "Generic (PLEG): container finished" podID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerID="9930de1860ed81baca1284740a91f1a926cabad3190cfd293f4f71662b2a4566" exitCode=60 Oct 30 03:15:02.741605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.741512 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerDied","Data":"9930de1860ed81baca1284740a91f1a926cabad3190cfd293f4f71662b2a4566"} Oct 30 03:15:02.741736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.741723 4560 scope.go:122] "RemoveContainer" containerID="9930de1860ed81baca1284740a91f1a926cabad3190cfd293f4f71662b2a4566" Oct 30 03:15:02.821238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:02.821188 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" podStartSLOduration=2.821173496 podStartE2EDuration="2.821173496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.819454173 +0000 UTC m=+730.804574039" watchObservedRunningTime="2026-10-30 03:15:02.821173496 +0000 UTC m=+730.806293386" Oct 30 03:15:03.748335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.748300 4560 generic.go:356] "Generic (PLEG): container finished" podID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerID="c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876" exitCode=60 Oct 30 03:15:03.748846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.748378 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerDied","Data":"c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876"} Oct 30 03:15:03.748846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.748467 4560 scope.go:122] "RemoveContainer" containerID="9930de1860ed81baca1284740a91f1a926cabad3190cfd293f4f71662b2a4566" Oct 30 03:15:03.748846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.748719 4560 scope.go:122] "RemoveContainer" containerID="c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876" Oct 30 03:15:03.749028 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:03.749003 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr_odh-ai-gateway-infra(b269495f-a153-4dd7-9a7b-bc2d702cc929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" Oct 30 03:15:03.750343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.750324 4560 scope.go:122] "RemoveContainer" containerID="8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255" Oct 30 03:15:03.750609 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:03.750588 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s_odh-ai-gateway-infra(60ff2498-3b3b-40d2-a3f8-a076c4cf11d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" Oct 30 03:15:03.751886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.751867 4560 generic.go:356] "Generic (PLEG): container finished" podID="36a9d734-958f-471e-a36c-3a953e288997" containerID="04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce" exitCode=60 Oct 30 03:15:03.751981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.751894 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerDied","Data":"04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce"} Oct 30 03:15:03.752121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.752106 4560 scope.go:122] "RemoveContainer" containerID="04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce" Oct 30 03:15:03.752376 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:03.752351 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62_odh-ai-gateway-infra(36a9d734-958f-471e-a36c-3a953e288997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" podUID="36a9d734-958f-471e-a36c-3a953e288997" Oct 30 03:15:03.759717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:03.759700 4560 scope.go:122] "RemoveContainer" containerID="4dd5a5d156692149c29bad06eec8653865f7d5546e067dca0e13383638ec7ac4" Oct 30 03:15:04.760535 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:04.760480 4560 scope.go:122] "RemoveContainer" containerID="c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876" Oct 30 03:15:04.761029 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:04.760764 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr_odh-ai-gateway-infra(b269495f-a153-4dd7-9a7b-bc2d702cc929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" Oct 30 03:15:04.762381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:04.762361 4560 scope.go:122] "RemoveContainer" containerID="04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce" Oct 30 03:15:04.762655 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:04.762632 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62_odh-ai-gateway-infra(36a9d734-958f-471e-a36c-3a953e288997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" podUID="36a9d734-958f-471e-a36c-3a953e288997" Oct 30 03:15:09.419751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:09.419709 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:15:09.432692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:09.432658 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:15:09.799192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:09.799164 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:15:11.751089 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:11.751054 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:15:13.145648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:13.145613 4560 scope.go:122] "RemoveContainer" containerID="8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255" Oct 30 03:15:13.146171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:13.145906 4560 scope.go:122] "RemoveContainer" containerID="04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce" Oct 30 03:15:13.810505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:13.810416 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerStarted","Data":"5f3a02cb122a7aa0ca4af145de481cae2c1bc1189e298019adcd1d9d0f0bfafa"} Oct 30 03:15:13.812343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:13.812322 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerStarted","Data":"84f56109276e46b108fdb1fe41c0c2486b70bda2cfd44006c11f867336a5fb05"} Oct 30 03:15:13.895200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:13.895144 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" podStartSLOduration=13.895129547 podStartE2EDuration="13.895129547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:13.893410573 +0000 UTC m=+741.878530462" watchObservedRunningTime="2026-10-30 03:15:13.895129547 +0000 UTC m=+741.880249434" Oct 30 03:15:14.819566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.819502 4560 generic.go:356] "Generic (PLEG): container finished" podID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerID="84f56109276e46b108fdb1fe41c0c2486b70bda2cfd44006c11f867336a5fb05" exitCode=60 Oct 30 03:15:14.820040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.819584 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerDied","Data":"84f56109276e46b108fdb1fe41c0c2486b70bda2cfd44006c11f867336a5fb05"} Oct 30 03:15:14.820040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.819660 4560 scope.go:122] "RemoveContainer" containerID="8f4d42773de3abcfd659265198c0e475700fe9c389d58e91ea19f31cc57e8255" Oct 30 03:15:14.820040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.820027 4560 scope.go:122] "RemoveContainer" containerID="84f56109276e46b108fdb1fe41c0c2486b70bda2cfd44006c11f867336a5fb05" Oct 30 03:15:14.820312 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:14.820290 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s_odh-ai-gateway-infra(60ff2498-3b3b-40d2-a3f8-a076c4cf11d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" Oct 30 03:15:14.821938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.821918 4560 generic.go:356] "Generic (PLEG): container finished" podID="36a9d734-958f-471e-a36c-3a953e288997" containerID="5f3a02cb122a7aa0ca4af145de481cae2c1bc1189e298019adcd1d9d0f0bfafa" exitCode=60 Oct 30 03:15:14.822028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.821972 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerDied","Data":"5f3a02cb122a7aa0ca4af145de481cae2c1bc1189e298019adcd1d9d0f0bfafa"} Oct 30 03:15:14.822231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.822218 4560 scope.go:122] "RemoveContainer" containerID="5f3a02cb122a7aa0ca4af145de481cae2c1bc1189e298019adcd1d9d0f0bfafa" Oct 30 03:15:14.822426 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:14.822412 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62_odh-ai-gateway-infra(36a9d734-958f-471e-a36c-3a953e288997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" podUID="36a9d734-958f-471e-a36c-3a953e288997" Oct 30 03:15:14.831104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.831088 4560 scope.go:122] "RemoveContainer" containerID="04088ebc1e2695628d8ed887254c9dd0f99d5e8cc81a7de216682b36c9080cce" Oct 30 03:15:14.863129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.863103 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:14.902661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:14.902629 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:15.145037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:15.144957 4560 scope.go:122] "RemoveContainer" containerID="c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876" Oct 30 03:15:15.831034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:15.831008 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerStarted","Data":"bbda12fa0ee8779e6df71867b62e20bf15cd86b0ebf23f934d01edf070d215ac"} Oct 30 03:15:16.054365 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.054335 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:16.057388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.057370 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:16.120630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.120552 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\") pod \"36a9d734-958f-471e-a36c-3a953e288997\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " Oct 30 03:15:16.120801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.120687 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\" (UniqueName: \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\") pod \"36a9d734-958f-471e-a36c-3a953e288997\" (UID: \"36a9d734-958f-471e-a36c-3a953e288997\") " Oct 30 03:15:16.120801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.120721 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\") pod \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " Oct 30 03:15:16.120801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.120757 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\" (UniqueName: \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\") pod \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\" (UID: \"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8\") " Oct 30 03:15:16.121043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.121011 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca" pod "36a9d734-958f-471e-a36c-3a953e288997" (UID: "36a9d734-958f-471e-a36c-3a953e288997"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:16.121134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.121113 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d734-958f-471e-a36c-3a953e288997-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.121195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.121132 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca" pod "60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" (UID: "60ff2498-3b3b-40d2-a3f8-a076c4cf11d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:16.122863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.122841 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74" pod "36a9d734-958f-471e-a36c-3a953e288997" (UID: "36a9d734-958f-471e-a36c-3a953e288997"). InnerVolumeSpecName "kube-api-access-tpd74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:15:16.123041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.123022 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px" pod "60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" (UID: "60ff2498-3b3b-40d2-a3f8-a076c4cf11d8"). InnerVolumeSpecName "kube-api-access-q75px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:15:16.222432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.222404 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpd74\" (UniqueName: \"kubernetes.io/projected/36a9d734-958f-471e-a36c-3a953e288997-kube-api-access-tpd74\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.222432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.222431 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.222639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.222440 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75px\" (UniqueName: \"kubernetes.io/projected/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8-kube-api-access-q75px\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.839684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.839650 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" Oct 30 03:15:16.840126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.839683 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62" event={"ID":"36a9d734-958f-471e-a36c-3a953e288997","Type":"ContainerDied","Data":"f8bcfc414da4dedfe9ae50b9dce0a77dfe4f097618434daa1865f301fbcc5891"} Oct 30 03:15:16.840126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.839794 4560 scope.go:122] "RemoveContainer" containerID="5f3a02cb122a7aa0ca4af145de481cae2c1bc1189e298019adcd1d9d0f0bfafa" Oct 30 03:15:16.842303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.842280 4560 generic.go:356] "Generic (PLEG): container finished" podID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerID="bbda12fa0ee8779e6df71867b62e20bf15cd86b0ebf23f934d01edf070d215ac" exitCode=60 Oct 30 03:15:16.842427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.842361 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerDied","Data":"bbda12fa0ee8779e6df71867b62e20bf15cd86b0ebf23f934d01edf070d215ac"} Oct 30 03:15:16.842807 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.842785 4560 scope.go:122] "RemoveContainer" containerID="bbda12fa0ee8779e6df71867b62e20bf15cd86b0ebf23f934d01edf070d215ac" Oct 30 03:15:16.843097 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:16.843067 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr_odh-ai-gateway-infra(b269495f-a153-4dd7-9a7b-bc2d702cc929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" Oct 30 03:15:16.843760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.843742 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" Oct 30 03:15:16.843855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.843739 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s" event={"ID":"60ff2498-3b3b-40d2-a3f8-a076c4cf11d8","Type":"ContainerDied","Data":"c4c9a90acd94e4b962d0efbc5e7adddb1f3d9b341dde88744e4ab380665e922c"} Oct 30 03:15:16.850085 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.850068 4560 scope.go:122] "RemoveContainer" containerID="c15bb2ee2ba2cad2b5ec2585047ea15ea39247276f7619003b668bc32e469876" Oct 30 03:15:16.862419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.862401 4560 scope.go:122] "RemoveContainer" containerID="84f56109276e46b108fdb1fe41c0c2486b70bda2cfd44006c11f867336a5fb05" Oct 30 03:15:16.920159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.920128 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:16.934596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.934567 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:16.940097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.940070 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888835-96v8s"] Oct 30 03:15:16.975785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.975752 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:16.981211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:16.981182 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888835-r9j62"] Oct 30 03:15:17.992505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:17.992476 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:18.038583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.038545 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\") pod \"b269495f-a153-4dd7-9a7b-bc2d702cc929\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " Oct 30 03:15:18.038750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.038644 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\" (UniqueName: \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\") pod \"b269495f-a153-4dd7-9a7b-bc2d702cc929\" (UID: \"b269495f-a153-4dd7-9a7b-bc2d702cc929\") " Oct 30 03:15:18.039001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.038968 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca" pod "b269495f-a153-4dd7-9a7b-bc2d702cc929" (UID: "b269495f-a153-4dd7-9a7b-bc2d702cc929"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:18.040916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.040892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9" pod "b269495f-a153-4dd7-9a7b-bc2d702cc929" (UID: "b269495f-a153-4dd7-9a7b-bc2d702cc929"). InnerVolumeSpecName "kube-api-access-m6cf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:15:18.140138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.140059 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269495f-a153-4dd7-9a7b-bc2d702cc929-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:18.140138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.140091 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6cf9\" (UniqueName: \"kubernetes.io/projected/b269495f-a153-4dd7-9a7b-bc2d702cc929-kube-api-access-m6cf9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:15:18.151638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.151606 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a9d734-958f-471e-a36c-3a953e288997" path="/var/lib/kubelet/pods/36a9d734-958f-471e-a36c-3a953e288997/volumes" Oct 30 03:15:18.152010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.151998 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" path="/var/lib/kubelet/pods/60ff2498-3b3b-40d2-a3f8-a076c4cf11d8/volumes" Oct 30 03:15:18.859689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.859653 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" Oct 30 03:15:18.859689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.859661 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr" event={"ID":"b269495f-a153-4dd7-9a7b-bc2d702cc929","Type":"ContainerDied","Data":"d4be5da845000d90327935777c3e173834e6914c428443fc9896784cf281d206"} Oct 30 03:15:18.859943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.859716 4560 scope.go:122] "RemoveContainer" containerID="bbda12fa0ee8779e6df71867b62e20bf15cd86b0ebf23f934d01edf070d215ac" Oct 30 03:15:18.914322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.914292 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:18.927459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:18.927429 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888835-mwkfr"] Oct 30 03:15:20.152528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:20.152477 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" path="/var/lib/kubelet/pods/b269495f-a153-4dd7-9a7b-bc2d702cc929/volumes" Oct 30 03:15:22.737508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.737473 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738119 4560 cpu_manager.go:414] "containerMap: removing container" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738144 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738161 4560 cpu_manager.go:414] "containerMap: removing container" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738168 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738182 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738190 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738213 4560 cpu_manager.go:414] "containerMap: removing container" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738219 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738244 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738251 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738271 4560 cpu_manager.go:414] "containerMap: removing container" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738278 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738364 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738375 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738384 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738398 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738406 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738414 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.738685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.738428 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:22.743473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.743451 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.761038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.761007 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:15:22.770972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.770938 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"e2e-conflict-b-9efa19e9-kserve-self-signed-certs\"" Oct 30 03:15:22.791906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.791874 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.792052 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.791921 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4h5n\" (UniqueName: \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.792114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.792089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.792165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.792114 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.792165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.792140 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.792267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.792164 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893150 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893197 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893223 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893246 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893346 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893373 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4h5n\" (UniqueName: \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893641 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893678 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.893900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.893880 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.895718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.895697 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.896200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.896178 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:22.915889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:22.915864 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4h5n\" (UniqueName: \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\") pod \"e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:23.059104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:23.059016 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:23.360349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:23.360323 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:15:23.361565 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:23.361535 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881dee11_1d75_4f06_a05f_7d2fcdb3c01a.slice/crio-af97baf2cc96e8973a877bd1837d39bafa891a8317c31a5dc3e0512b8914edfc WatchSource:0}: Error finding container af97baf2cc96e8973a877bd1837d39bafa891a8317c31a5dc3e0512b8914edfc: Status 404 returned error can't find the container with id af97baf2cc96e8973a877bd1837d39bafa891a8317c31a5dc3e0512b8914edfc Oct 30 03:15:23.889014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:23.888983 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" event={"ID":"881dee11-1d75-4f06-a05f-7d2fcdb3c01a","Type":"ContainerStarted","Data":"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc"} Oct 30 03:15:23.889396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:23.889020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" event={"ID":"881dee11-1d75-4f06-a05f-7d2fcdb3c01a","Type":"ContainerStarted","Data":"af97baf2cc96e8973a877bd1837d39bafa891a8317c31a5dc3e0512b8914edfc"} Oct 30 03:15:23.930213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:23.930164 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" podStartSLOduration=1.930148125 podStartE2EDuration="1.930148125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:23.928564416 +0000 UTC m=+751.913684305" watchObservedRunningTime="2026-10-30 03:15:23.930148125 +0000 UTC m=+751.915268012" Oct 30 03:15:24.059486 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:24.059454 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:33.060236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:33.060194 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:33.073184 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:33.073153 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:33.954642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:33.954558 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:15:36.458382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458349 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458793 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458808 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269495f-a153-4dd7-9a7b-bc2d702cc929" containerName="cleanup" Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458839 4560 cpu_manager.go:414] "containerMap: removing container" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458845 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458863 4560 cpu_manager.go:414] "containerMap: removing container" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:36.458914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458868 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:36.459288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458936 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ff2498-3b3b-40d2-a3f8-a076c4cf11d8" containerName="cleanup" Oct 30 03:15:36.459288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.458950 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a9d734-958f-471e-a36c-3a953e288997" containerName="cleanup" Oct 30 03:15:36.462471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.462447 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.481922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.481888 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 03:15:36.492322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.492296 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"distinct-w4-8d80b1-kserve-self-signed-certs\"" Oct 30 03:15:36.542164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542132 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.542332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542179 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.542332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542269 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.542332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542291 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ws8\" (UniqueName: \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.542428 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542337 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.542428 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.542375 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643351 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643430 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ws8\" (UniqueName: \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643542 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643596 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643759 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.643978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.643803 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.644158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.644132 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.644323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.644185 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.644440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.644416 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.650473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.647140 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.651296 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.651267 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.664245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.664218 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ws8\" (UniqueName: \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\") pod \"distinct-w4-8d80b1-kserve-9fb458976-grpsk\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:36.775906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:36.775821 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:37.077817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:37.077790 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 03:15:37.080090 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:37.080064 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd833a78d_8e47_45ea_9e16_0b503352a97c.slice/crio-6d2be694dea87242818096c3c35faf8ae041f257497437ab8283451c1da7aa69 WatchSource:0}: Error finding container 6d2be694dea87242818096c3c35faf8ae041f257497437ab8283451c1da7aa69: Status 404 returned error can't find the container with id 6d2be694dea87242818096c3c35faf8ae041f257497437ab8283451c1da7aa69 Oct 30 03:15:37.968196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:37.968156 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" event={"ID":"d833a78d-8e47-45ea-9e16-0b503352a97c","Type":"ContainerStarted","Data":"66fa9abf1fffe51da51235f39e80d57411e9e7dc9a21da367689da708716554f"} Oct 30 03:15:37.968684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:37.968205 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" event={"ID":"d833a78d-8e47-45ea-9e16-0b503352a97c","Type":"ContainerStarted","Data":"6d2be694dea87242818096c3c35faf8ae041f257497437ab8283451c1da7aa69"} Oct 30 03:15:38.007727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:38.007678 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" podStartSLOduration=2.007664549 podStartE2EDuration="2.007664549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:38.006770687 +0000 UTC m=+765.991890574" watchObservedRunningTime="2026-10-30 03:15:38.007664549 +0000 UTC m=+765.992784459" Oct 30 03:15:38.776349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:38.776315 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:41.750149 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:15:41.750105 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:15:43.055001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.054969 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:15:43.059611 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.059581 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.076100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.076079 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2d5b09-openshift-default-dockercfg-hpf84\"" Oct 30 03:15:43.091722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.091691 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:15:43.213856 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.213820 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.213888 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.213951 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.213971 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.213992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.214011 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.214070 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.214118 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hpc\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.214328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.214227 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.315689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315591 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.315689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315682 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.315929 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315719 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.315929 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315761 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315939 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.315987 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316016 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316193 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316049 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316193 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316079 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hpc\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316416 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316496 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316582 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.316664 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.316637 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.317149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.317128 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.318218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.318194 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.318422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.318406 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.336055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.336026 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.336248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.336229 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hpc\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.373612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.373583 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:43.710497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.710464 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:15:43.711336 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:43.711299 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f799c8_bc83_436d_b06d_01d6bc8695c6.slice/crio-997b981536fcf55cb39c56f777633fda25ef2495ed0cc80b751906b1c2019f7e WatchSource:0}: Error finding container 997b981536fcf55cb39c56f777633fda25ef2495ed0cc80b751906b1c2019f7e: Status 404 returned error can't find the container with id 997b981536fcf55cb39c56f777633fda25ef2495ed0cc80b751906b1c2019f7e Oct 30 03:15:43.713861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.713830 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:43.713940 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.713893 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:43.713940 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:43.713922 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:44.001212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.001103 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" event={"ID":"d4f799c8-bc83-436d-b06d-01d6bc8695c6","Type":"ContainerStarted","Data":"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639"} Oct 30 03:15:44.001212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.001170 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" event={"ID":"d4f799c8-bc83-436d-b06d-01d6bc8695c6","Type":"ContainerStarted","Data":"997b981536fcf55cb39c56f777633fda25ef2495ed0cc80b751906b1c2019f7e"} Oct 30 03:15:44.049557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.049474 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podStartSLOduration=1.049454036 podStartE2EDuration="1.049454036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:44.047816741 +0000 UTC m=+772.032936647" watchObservedRunningTime="2026-10-30 03:15:44.049454036 +0000 UTC m=+772.034573925" Oct 30 03:15:44.374685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.374587 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:44.375400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.375371 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:44.375503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:44.375454 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:45.230970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:45.230928 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:15:45.375541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:45.374904 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:45.375541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:45.374968 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:46.374969 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.374931 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:46.375151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.374994 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:46.782557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.778344 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:46.786554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.784949 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:15:46.803496 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.803435 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:46.873542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.872636 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 03:15:46.893574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.893500 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.897941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.893572 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 03:15:46.913442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.913416 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e44313-openshift-default-dockercfg-tlmnm\"" Oct 30 03:15:46.979902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.979860 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.979914 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980017 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980053 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzxh\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980182 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980222 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980306 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:46.980372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:46.980332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.028949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.028914 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 03:15:47.081324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081233 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081286 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzxh\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081333 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081366 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081417 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081449 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081576 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081631 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081688 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.081891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.081844 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.082134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.082114 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.082334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.082286 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.082450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.082433 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.082506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.082444 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.083794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.083756 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.084610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.084583 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.115566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.115535 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzxh\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.115965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.115944 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.216678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.216640 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:47.375031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.374947 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:47.375031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.375007 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:47.562542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.559599 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 03:15:47.581120 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:47.581084 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61014647_b515_45f6_886c_87b102274c29.slice/crio-95c360b784a66c1caff650ee573d66974b1cea6fccec13c564d0e682595d7e9f WatchSource:0}: Error finding container 95c360b784a66c1caff650ee573d66974b1cea6fccec13c564d0e682595d7e9f: Status 404 returned error can't find the container with id 95c360b784a66c1caff650ee573d66974b1cea6fccec13c564d0e682595d7e9f Oct 30 03:15:47.584132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.584069 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:47.584262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.584244 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:47.584359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:47.584336 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:48.025097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.025060 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" event={"ID":"61014647-b515-45f6-886c-87b102274c29","Type":"ContainerStarted","Data":"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd"} Oct 30 03:15:48.025622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.025139 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" event={"ID":"61014647-b515-45f6-886c-87b102274c29","Type":"ContainerStarted","Data":"95c360b784a66c1caff650ee573d66974b1cea6fccec13c564d0e682595d7e9f"} Oct 30 03:15:48.076543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.076179 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podStartSLOduration=2.076160345 podStartE2EDuration="2.076160345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:48.073308198 +0000 UTC m=+776.058428099" watchObservedRunningTime="2026-10-30 03:15:48.076160345 +0000 UTC m=+776.061280234" Oct 30 03:15:48.217544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.217492 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:48.218932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.218903 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:48.219036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.218958 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:48.391602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.377960 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:48.391602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:48.378032 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:49.217336 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:49.217302 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:49.217730 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:49.217356 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:49.388589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:49.387997 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:49.388589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:49.388077 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:50.217477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:50.217437 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:50.217983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:50.217492 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:50.374539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:50.374119 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:15:50.374539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:50.374180 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 03:15:51.217296 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:51.217267 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:51.217460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:51.217313 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:51.379094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:51.379059 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:51.379584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:51.379360 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:51.380059 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:51.380044 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:15:52.216984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:52.216949 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:52.217167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:52.217011 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:53.217867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:53.217832 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:53.218256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:53.217882 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:54.217215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:54.217179 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:15:54.217363 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:54.217232 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 03:15:55.692195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:55.692149 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:55.692664 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:55.692639 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:55.693600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:55.693576 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 03:15:59.067037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.067005 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:15:59.072427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.072406 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.087486 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.087458 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d80b1-openshift-default-dockercfg-22948\"" Oct 30 03:15:59.119224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.119191 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:15:59.138594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138567 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138607 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138642 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138742 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcjc\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138787 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138882 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138917 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.138962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138936 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.139032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.138984 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240378 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240421 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240590 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240776 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240755 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240783 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240793 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240894 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcjc\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.240957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240941 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.241111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240994 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.241111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.241066 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.241610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.240849 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.241610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.241295 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.241610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.241472 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.243262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.243241 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.244062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.244044 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.262353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.262330 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.262715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.262691 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcjc\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.389613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.389530 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:15:59.707424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.707389 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:15:59.708917 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:15:59.708891 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3919fa0_4578_413f_a771_20facbfb73ba.slice/crio-4257f5c678a1e76e3418d1839967e826fcc9fa4e5261178d8317a62db6f063d6 WatchSource:0}: Error finding container 4257f5c678a1e76e3418d1839967e826fcc9fa4e5261178d8317a62db6f063d6: Status 404 returned error can't find the container with id 4257f5c678a1e76e3418d1839967e826fcc9fa4e5261178d8317a62db6f063d6 Oct 30 03:15:59.711333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.711302 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:59.711426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.711365 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:15:59.711426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:15:59.711393 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:16:00.126850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.126768 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" event={"ID":"d3919fa0-4578-413f-a771-20facbfb73ba","Type":"ContainerStarted","Data":"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b"} Oct 30 03:16:00.126850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.126851 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" event={"ID":"d3919fa0-4578-413f-a771-20facbfb73ba","Type":"ContainerStarted","Data":"4257f5c678a1e76e3418d1839967e826fcc9fa4e5261178d8317a62db6f063d6"} Oct 30 03:16:00.180911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.180859 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podStartSLOduration=1.180847032 podStartE2EDuration="1.180847032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:00.179436221 +0000 UTC m=+788.164556109" watchObservedRunningTime="2026-10-30 03:16:00.180847032 +0000 UTC m=+788.165966919" Oct 30 03:16:00.287770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.287730 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" podUID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" containerName="main" containerID="cri-o://cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc" gracePeriod=45 Oct 30 03:16:00.389892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.389807 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:16:00.391546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.391508 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:00.391694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.391566 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:00.603715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.603689 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:16:00.780088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780050 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780130 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780275 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\" (UniqueName: \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780306 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780362 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780349 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:00.780405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780392 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:00.780405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780399 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\" (UniqueName: \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\") pod \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\" (UID: \"881dee11-1d75-4f06-a05f-7d2fcdb3c01a\") " Oct 30 03:16:00.780618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780557 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:00.780844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780824 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:00.780844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780844 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:00.780950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.780853 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:00.782564 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.782539 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:00.782976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.782954 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:16:00.783057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.782971 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n" pod "881dee11-1d75-4f06-a05f-7d2fcdb3c01a" (UID: "881dee11-1d75-4f06-a05f-7d2fcdb3c01a"). InnerVolumeSpecName "kube-api-access-z4h5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:16:00.882304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.882261 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:00.882304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.882299 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4h5n\" (UniqueName: \"kubernetes.io/projected/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-kube-api-access-z4h5n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:00.882512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:00.882314 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881dee11-1d75-4f06-a05f-7d2fcdb3c01a-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:01.133537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.133435 4560 generic.go:356] "Generic (PLEG): container finished" podID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" containerID="cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc" exitCode=0 Oct 30 03:16:01.133537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.133484 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" Oct 30 03:16:01.134041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.133552 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" event={"ID":"881dee11-1d75-4f06-a05f-7d2fcdb3c01a","Type":"ContainerDied","Data":"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc"} Oct 30 03:16:01.134041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.133589 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2" event={"ID":"881dee11-1d75-4f06-a05f-7d2fcdb3c01a","Type":"ContainerDied","Data":"af97baf2cc96e8973a877bd1837d39bafa891a8317c31a5dc3e0512b8914edfc"} Oct 30 03:16:01.134041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.133606 4560 scope.go:122] "RemoveContainer" containerID="cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc" Oct 30 03:16:01.142978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.142961 4560 scope.go:122] "RemoveContainer" containerID="cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc" Oct 30 03:16:01.143264 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:16:01.143243 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc\": container with ID starting with cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc not found: ID does not exist" containerID="cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc" Oct 30 03:16:01.143339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.143273 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc"} err="failed to get container status \"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc\": rpc error: code = NotFound desc = could not find container \"cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc\": container with ID starting with cc44bfbf61273c5917862863a5b84da8accf5416b2277522217eddc5ea185edc not found: ID does not exist" Oct 30 03:16:01.207225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.207195 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:16:01.219534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.219494 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-b-9efa19e9-kserve-5474cf6b5b-2bvd2"] Oct 30 03:16:01.390156 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.390075 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:01.390156 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.390126 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:01.810772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:01.810725 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" podUID="859adab4-61bf-4461-b91f-927ceb60eb4f" containerName="main" containerID="cri-o://90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae" gracePeriod=45 Oct 30 03:16:02.116290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.116052 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:16:02.146555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.142241 4560 generic.go:356] "Generic (PLEG): container finished" podID="859adab4-61bf-4461-b91f-927ceb60eb4f" containerID="90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae" exitCode=0 Oct 30 03:16:02.146555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.142295 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" Oct 30 03:16:02.146555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.142291 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" event={"ID":"859adab4-61bf-4461-b91f-927ceb60eb4f","Type":"ContainerDied","Data":"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae"} Oct 30 03:16:02.146555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.142418 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf" event={"ID":"859adab4-61bf-4461-b91f-927ceb60eb4f","Type":"ContainerDied","Data":"dcc1ca5bb19507209d906507aadcfe81ad66964f436b86958f816eef4608ad66"} Oct 30 03:16:02.146555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.142437 4560 scope.go:122] "RemoveContainer" containerID="90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae" Oct 30 03:16:02.169724 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.166432 4560 scope.go:122] "RemoveContainer" containerID="90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae" Oct 30 03:16:02.180553 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:16:02.178668 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae\": container with ID starting with 90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae not found: ID does not exist" containerID="90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae" Oct 30 03:16:02.180553 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.178713 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae"} err="failed to get container status \"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae\": rpc error: code = NotFound desc = could not find container \"90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae\": container with ID starting with 90b703993d25f6b3db2dafdcea1b099d3096ccd64f85c46b75dac28ea76555ae not found: ID does not exist" Oct 30 03:16:02.184549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.181502 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" path="/var/lib/kubelet/pods/881dee11-1d75-4f06-a05f-7d2fcdb3c01a/volumes" Oct 30 03:16:02.299324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299290 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299527 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299385 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299527 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299430 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299527 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299465 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\" (UniqueName: \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299527 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299510 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299561 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\") pod \"859adab4-61bf-4461-b91f-927ceb60eb4f\" (UID: \"859adab4-61bf-4461-b91f-927ceb60eb4f\") " Oct 30 03:16:02.299690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299613 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:02.299850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.299828 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:02.300208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.300041 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.300208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.300063 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.300467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.300408 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:02.302177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.302151 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "kube-api-access-76b7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:16:02.302362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.302334 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:02.302792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.302768 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs" pod "859adab4-61bf-4461-b91f-927ceb60eb4f" (UID: "859adab4-61bf-4461-b91f-927ceb60eb4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:16:02.389909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.389824 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:02.389909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.389871 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:02.401646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.401621 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859adab4-61bf-4461-b91f-927ceb60eb4f-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.401646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.401646 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76b7f\" (UniqueName: \"kubernetes.io/projected/859adab4-61bf-4461-b91f-927ceb60eb4f-kube-api-access-76b7f\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.401784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.401655 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.401784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.401665 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859adab4-61bf-4461-b91f-927ceb60eb4f-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:16:02.516336 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.516305 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:16:02.519726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:02.519700 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/e2e-conflict-a-9efa19e9-kserve-5dbdc65ccd-4rvtf"] Oct 30 03:16:03.390165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:03.390131 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:03.390593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:03.390187 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:04.164540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:04.163018 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859adab4-61bf-4461-b91f-927ceb60eb4f" path="/var/lib/kubelet/pods/859adab4-61bf-4461-b91f-927ceb60eb4f/volumes" Oct 30 03:16:04.390773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:04.390734 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:04.391259 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:04.390801 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:05.390153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:05.390114 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:05.390339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:05.390178 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:06.390358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:06.390320 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 03:16:06.390761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:06.390387 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 03:16:07.394901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:07.394870 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:16:07.395313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:07.395216 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:16:07.395872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:16:07.395855 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:16:11.748594 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:16:11.748555 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:16:41.749137 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:16:41.749098 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:17:11.750468 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:17:11.750335 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:17:41.747658 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:17:41.747622 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:18:11.748898 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:18:11.748863 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:18:33.497194 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:33.497155 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:18:33.497628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:33.497372 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" containerID="cri-o://a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b" gracePeriod=30 Oct 30 03:18:38.746586 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.746559 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:18:38.874020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.873940 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.873986 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874040 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874081 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874187 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874215 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:18:38.874386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874253 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874282 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874312 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874379 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\") pod \"1491d054-52a8-4690-b522-dd13c03fab34\" (UID: \"1491d054-52a8-4690-b522-dd13c03fab34\") " Oct 30 03:18:38.874653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874503 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:18:38.874653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874575 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:18:38.874653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874480 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:18:38.874653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874602 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:18:38.874832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874787 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.874832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874833 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.874986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874846 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1491d054-52a8-4690-b522-dd13c03fab34-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.874986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874859 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.874986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.874872 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.876942 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.876920 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:18:38.877058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.876937 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:38.877058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.877025 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:18:38.877232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.877214 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22" pod "1491d054-52a8-4690-b522-dd13c03fab34" (UID: "1491d054-52a8-4690-b522-dd13c03fab34"). InnerVolumeSpecName "kube-api-access-jtd22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:38.975747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.975716 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1491d054-52a8-4690-b522-dd13c03fab34-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.975747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.975743 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd22\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-kube-api-access-jtd22\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.975747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.975753 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1491d054-52a8-4690-b522-dd13c03fab34-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.975976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.975762 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1491d054-52a8-4690-b522-dd13c03fab34-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:18:38.994904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.994876 4560 generic.go:356] "Generic (PLEG): container finished" podID="1491d054-52a8-4690-b522-dd13c03fab34" containerID="a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b" exitCode=0 Oct 30 03:18:38.995058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.994923 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" Oct 30 03:18:38.995058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.994965 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" event={"ID":"1491d054-52a8-4690-b522-dd13c03fab34","Type":"ContainerDied","Data":"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b"} Oct 30 03:18:38.995058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.995039 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc" event={"ID":"1491d054-52a8-4690-b522-dd13c03fab34","Type":"ContainerDied","Data":"6cd11055d42e76411da86699367dbb4ee12f2e6a9b621ecb561601c6cc4252d5"} Oct 30 03:18:38.995162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:38.995060 4560 scope.go:122] "RemoveContainer" containerID="a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b" Oct 30 03:18:39.004747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:39.004730 4560 scope.go:122] "RemoveContainer" containerID="a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b" Oct 30 03:18:39.005002 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:18:39.004983 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b\": container with ID starting with a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b not found: ID does not exist" containerID="a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b" Oct 30 03:18:39.005041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:39.005014 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b"} err="failed to get container status \"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b\": rpc error: code = NotFound desc = could not find container \"a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b\": container with ID starting with a00354ffadecaefbea550adf32d02857f8123b0f21cef9512cba4181d7f9562b not found: ID does not exist" Oct 30 03:18:39.046996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:39.046961 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:18:39.050280 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:39.050259 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpfhc"] Oct 30 03:18:40.152362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:40.152329 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1491d054-52a8-4690-b522-dd13c03fab34" path="/var/lib/kubelet/pods/1491d054-52a8-4690-b522-dd13c03fab34/volumes" Oct 30 03:18:41.748856 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:18:41.748820 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:18:52.321579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:52.321549 4560 scope.go:122] "RemoveContainer" containerID="3ac929e3059d8baa15a1c0c19ec5373d4fc638d9d6b26f4225a5d0ce206f0d98" Oct 30 03:18:52.330387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:18:52.330368 4560 scope.go:122] "RemoveContainer" containerID="70c132910d83e1c24b25b7dc8fdc07b96013f186036ea66bd07cabe2c5d284e6" Oct 30 03:19:03.591861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.591822 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:19:03.592408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592388 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" Oct 30 03:19:03.592454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592409 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" Oct 30 03:19:03.592454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592441 4560 cpu_manager.go:414] "containerMap: removing container" podUID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" containerName="main" Oct 30 03:19:03.592454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592446 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" containerName="main" Oct 30 03:19:03.592599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592467 4560 cpu_manager.go:414] "containerMap: removing container" podUID="859adab4-61bf-4461-b91f-927ceb60eb4f" containerName="main" Oct 30 03:19:03.592599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592472 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859adab4-61bf-4461-b91f-927ceb60eb4f" containerName="main" Oct 30 03:19:03.592599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592554 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1491d054-52a8-4690-b522-dd13c03fab34" containerName="istio-proxy" Oct 30 03:19:03.592599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592566 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="881dee11-1d75-4f06-a05f-7d2fcdb3c01a" containerName="main" Oct 30 03:19:03.592599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.592574 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="859adab4-61bf-4461-b91f-927ceb60eb4f" containerName="main" Oct 30 03:19:03.597504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.597481 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.613583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.613559 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fqfn6\"" Oct 30 03:19:03.628571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.628544 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:19:03.737437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737382 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737476 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737561 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737586 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737610 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737625 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9lb\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737736 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737785 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.737877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.737856 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.838832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838798 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838844 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838866 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838881 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9lb\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838903 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838940 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.838975 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839065 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839131 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839321 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839501 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839479 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839565 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839645 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.839707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.839643 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.841821 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.841797 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.841946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.841911 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.858813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.858784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.858919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.858881 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9lb\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gkhfx\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:03.910007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:03.909975 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:04.229793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:04.229769 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:19:04.231463 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:19:04.231439 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2ef183_6a0f_4429_bd85_0c4d5f0d807a.slice/crio-c53ae1738eea43e86128c7a5cbc0879fc61f99ce15d7f7b095346b9984e927bf WatchSource:0}: Error finding container c53ae1738eea43e86128c7a5cbc0879fc61f99ce15d7f7b095346b9984e927bf: Status 404 returned error can't find the container with id c53ae1738eea43e86128c7a5cbc0879fc61f99ce15d7f7b095346b9984e927bf Oct 30 03:19:04.233781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:04.233749 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:04.233859 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:04.233811 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:04.233859 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:04.233838 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:05.135949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.135904 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" event={"ID":"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a","Type":"ContainerStarted","Data":"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22"} Oct 30 03:19:05.136452 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.136000 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" event={"ID":"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a","Type":"ContainerStarted","Data":"c53ae1738eea43e86128c7a5cbc0879fc61f99ce15d7f7b095346b9984e927bf"} Oct 30 03:19:05.206392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.206326 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podStartSLOduration=2.20630752 podStartE2EDuration="2.20630752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:05.204946539 +0000 UTC m=+973.190066430" watchObservedRunningTime="2026-10-30 03:19:05.20630752 +0000 UTC m=+973.191427408" Oct 30 03:19:05.911207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.911166 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:05.912612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.912586 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:05.912735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:05.912637 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:06.910676 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:06.910638 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:06.911076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:06.910694 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:07.911115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:07.911085 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:07.911591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:07.911560 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:08.910896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:08.910859 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:08.911176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:08.911148 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:09.910802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:09.910770 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:09.910977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:09.910823 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:10.915069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:10.915027 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:19:10.915595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:10.915094 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 03:19:11.749576 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:19:11.749542 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:19:12.681581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:12.681550 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:12.682084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:12.681938 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:12.683086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:12.683056 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:19:13.020404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.020368 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:19:13.025313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.025165 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.046954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.046923 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b132360f\"/\"openshift-service-ca.crt\"" Oct 30 03:19:13.047161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.046941 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b132360f\"/\"default-dockercfg-gjtjd\"" Oct 30 03:19:13.047277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.047038 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b132360f\"/\"e2e-lifecycle-model-b132360f-kserve-self-signed-certs\"" Oct 30 03:19:13.047373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.047110 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b132360f\"/\"kube-root-ca.crt\"" Oct 30 03:19:13.050961 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.050932 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:19:13.072128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072096 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.072287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072152 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.072346 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.072393 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072369 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgt8\" (UniqueName: \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.072482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.072637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.072511 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173641 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173686 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgt8\" (UniqueName: \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173734 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173753 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173869 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.173965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.173950 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.174195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.174150 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.174378 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.174348 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.175636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.175598 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.176826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.176805 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.178701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.178677 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.195533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.194423 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgt8\" (UniqueName: \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\") pod \"e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.343030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.342935 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:13.651053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:13.651024 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:19:13.652484 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:19:13.652444 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27db272_504d_4191_aeb2_2bc7b7d3b96a.slice/crio-89b39e498d29e755571242828b7a94a6a69f6ae82729bebddcdae6c65f25e5ce WatchSource:0}: Error finding container 89b39e498d29e755571242828b7a94a6a69f6ae82729bebddcdae6c65f25e5ce: Status 404 returned error can't find the container with id 89b39e498d29e755571242828b7a94a6a69f6ae82729bebddcdae6c65f25e5ce Oct 30 03:19:14.217780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:14.217741 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" event={"ID":"c27db272-504d-4191-aeb2-2bc7b7d3b96a","Type":"ContainerStarted","Data":"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a"} Oct 30 03:19:14.218281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:14.217794 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" event={"ID":"c27db272-504d-4191-aeb2-2bc7b7d3b96a","Type":"ContainerStarted","Data":"89b39e498d29e755571242828b7a94a6a69f6ae82729bebddcdae6c65f25e5ce"} Oct 30 03:19:14.259323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:14.259250 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" podStartSLOduration=2.259229514 podStartE2EDuration="2.259229514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:14.258260635 +0000 UTC m=+982.243380523" watchObservedRunningTime="2026-10-30 03:19:14.259229514 +0000 UTC m=+982.244349403" Oct 30 03:19:14.343344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:14.343314 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:18.657143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.657102 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:19:18.661019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.660998 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.694419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.694378 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:19:18.739359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739322 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739358 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739495 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739562 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739601 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrg8\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739632 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739672 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739725 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.739875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.739810 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.840590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840550 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.840798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840641 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.840798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840674 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.840798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840729 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.840798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840767 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840824 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrg8\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840860 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.840940 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.841205 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.841466 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.841502 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.841627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.841675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.841661 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.843212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.843188 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.843860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.843838 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.861947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.861920 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.864000 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.863976 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrg8\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fcd5v\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:18.991400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:18.991366 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:19.338683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:19.338646 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:19:19.338981 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:19:19.338954 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f93672c_456c_4549_819b_d09b89995cb0.slice/crio-675d7b31529fec87b9ef5f2c550a40ce8e40138e7f0501b983f135b5626c6d72 WatchSource:0}: Error finding container 675d7b31529fec87b9ef5f2c550a40ce8e40138e7f0501b983f135b5626c6d72: Status 404 returned error can't find the container with id 675d7b31529fec87b9ef5f2c550a40ce8e40138e7f0501b983f135b5626c6d72 Oct 30 03:19:19.341662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:19.341623 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:19.341753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:19.341709 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:19.341791 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:19.341754 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:20.276380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.276338 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" event={"ID":"8f93672c-456c-4549-819b-d09b89995cb0","Type":"ContainerStarted","Data":"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a"} Oct 30 03:19:20.276905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.276406 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" event={"ID":"8f93672c-456c-4549-819b-d09b89995cb0","Type":"ContainerStarted","Data":"675d7b31529fec87b9ef5f2c550a40ce8e40138e7f0501b983f135b5626c6d72"} Oct 30 03:19:20.334936 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.334886 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podStartSLOduration=2.334871587 podStartE2EDuration="2.334871587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:20.333157476 +0000 UTC m=+988.318277389" watchObservedRunningTime="2026-10-30 03:19:20.334871587 +0000 UTC m=+988.319991510" Oct 30 03:19:20.991604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.991563 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:20.993105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.993078 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:20.993248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:20.993147 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:21.992574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:21.992544 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:21.993029 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:21.992603 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:22.992508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:22.992470 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:22.992718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:22.992567 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:23.343884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:23.343802 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:23.386168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:23.386136 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:23.992461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:23.992426 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:23.992668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:23.992489 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:24.310687 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:24.310614 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:19:24.992734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:24.992699 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:24.992927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:24.992767 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:25.992188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:25.992151 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:25.992597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:25.992234 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:26.992534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:26.992487 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 03:19:26.992937 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:26.992580 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 03:19:28.001965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:28.001934 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:28.002566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:28.002282 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:28.003598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:28.003577 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:19:37.034381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:37.034348 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd"] Oct 30 03:19:37.047243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:37.047207 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e70622-hjsjd"] Oct 30 03:19:38.152800 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:38.152756 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f02e51b-d369-4460-a9a0-59eef0fc9282" path="/var/lib/kubelet/pods/0f02e51b-d369-4460-a9a0-59eef0fc9282/volumes" Oct 30 03:19:41.748947 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:19:41.748901 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:19:46.767639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.766678 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:19:46.779342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.778200 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.795955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.795923 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:19:46.805753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.805717 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3e4daf-openshift-default-dockercfg-wdh2q\"" Oct 30 03:19:46.869041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.868999 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpmn\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869048 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869068 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869193 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869294 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869345 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869458 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869479 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.869590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.869575 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971164 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971195 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971240 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpmn\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971264 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971280 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971306 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971390 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971432 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971650 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971714 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.971814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.971783 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.972099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.972075 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.972271 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.972249 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.973688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.973664 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.974014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.973989 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.991784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.991678 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpmn\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:46.991784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:46.991739 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\") pod \"e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:47.102981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:47.102889 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:47.439825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:47.439793 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:19:47.440475 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:19:47.440448 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec8c7_df43_4496_a6f2_3f6f83f93768.slice/crio-4c2f42887676af242c015907356aa18a45f52995ec0f1bfbc877c700f4c66aa0 WatchSource:0}: Error finding container 4c2f42887676af242c015907356aa18a45f52995ec0f1bfbc877c700f4c66aa0: Status 404 returned error can't find the container with id 4c2f42887676af242c015907356aa18a45f52995ec0f1bfbc877c700f4c66aa0 Oct 30 03:19:47.443209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:47.443180 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:47.443292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:47.443242 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:47.443292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:47.443272 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:48.431894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:48.431856 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerStarted","Data":"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805"} Oct 30 03:19:48.432389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:48.431920 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerStarted","Data":"4c2f42887676af242c015907356aa18a45f52995ec0f1bfbc877c700f4c66aa0"} Oct 30 03:19:48.490248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:48.490178 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podStartSLOduration=2.490158051 podStartE2EDuration="2.490158051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:48.48960987 +0000 UTC m=+1016.474729783" watchObservedRunningTime="2026-10-30 03:19:48.490158051 +0000 UTC m=+1016.475277941" Oct 30 03:19:49.103663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:49.103628 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:49.109934 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:49.109903 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:49.437074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:49.436985 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:49.438142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:49.438119 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:19:58.738310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.738245 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:19:58.762993 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.762959 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.772877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.771606 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:19:58.781224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.781197 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cc0859-openshift-default-dockercfg-8qzgs\"" Oct 30 03:19:58.916038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.915995 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.916325 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916303 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.916500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916485 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.916640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916623 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.916774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916760 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.916903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916890 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.917007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.916992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7v9\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.917160 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.917146 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:58.917322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:58.917304 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018486 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018552 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018637 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018659 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018699 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.018838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018733 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018903 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7v9\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.018968 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.019015 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.019049 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.019049 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.019646 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.019839 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.019726 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.020039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.020018 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.021456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.021434 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.021575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.021479 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.038300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.038277 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.038639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.038618 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7v9\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-slz65\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.088977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.088943 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:19:59.421953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.421928 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:19:59.422971 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:19:59.422942 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e01cfa_3e66_4646_8a24_4d6dedb6b70b.slice/crio-5d643e649101d7d9b87c02658565f1be6c9a6d470cbb4a102c9ef40957cdd692 WatchSource:0}: Error finding container 5d643e649101d7d9b87c02658565f1be6c9a6d470cbb4a102c9ef40957cdd692: Status 404 returned error can't find the container with id 5d643e649101d7d9b87c02658565f1be6c9a6d470cbb4a102c9ef40957cdd692 Oct 30 03:19:59.425268 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.425237 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:59.425337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.425305 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:59.425381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.425334 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:19:59.500358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.500321 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerStarted","Data":"ce66890336a2b600b9d5e0e976ca7d2f1da779b013ab8b4882401ccc082c6a04"} Oct 30 03:19:59.500542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.500394 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerStarted","Data":"5d643e649101d7d9b87c02658565f1be6c9a6d470cbb4a102c9ef40957cdd692"} Oct 30 03:19:59.544638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:19:59.544583 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podStartSLOduration=1.544568474 podStartE2EDuration="1.544568474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:59.542819252 +0000 UTC m=+1027.527939153" watchObservedRunningTime="2026-10-30 03:19:59.544568474 +0000 UTC m=+1027.529688365" Oct 30 03:20:00.089505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:00.089471 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:00.095258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:00.095233 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:00.505985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:00.505956 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:00.506996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:00.506975 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:08.582620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.582490 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-4d3b7e"] Oct 30 03:20:08.586593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.586568 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:08.613478 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.613446 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 03:20:08.614012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.613984 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-4d3b7e"] Oct 30 03:20:08.643074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.643037 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhtr\" (UniqueName: \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\") pod \"test-curl-7576-4d3b7e\" (UID: \"9b9c7653-dc3d-46d5-814b-43fcbbf08496\") " pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:08.744459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.744414 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhtr\" (UniqueName: \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\") pod \"test-curl-7576-4d3b7e\" (UID: \"9b9c7653-dc3d-46d5-814b-43fcbbf08496\") " pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:08.765804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.765764 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhtr\" (UniqueName: \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\") pod \"test-curl-7576-4d3b7e\" (UID: \"9b9c7653-dc3d-46d5-814b-43fcbbf08496\") " pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:08.905165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:08.905082 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:09.241692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:09.241663 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-4d3b7e"] Oct 30 03:20:09.243775 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:20:09.243744 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b9c7653_dc3d_46d5_814b_43fcbbf08496.slice/crio-25e0404f58f7fbbf80e1c6c587ebcb2882d00794003e79f242a6d038febc76f2 WatchSource:0}: Error finding container 25e0404f58f7fbbf80e1c6c587ebcb2882d00794003e79f242a6d038febc76f2: Status 404 returned error can't find the container with id 25e0404f58f7fbbf80e1c6c587ebcb2882d00794003e79f242a6d038febc76f2 Oct 30 03:20:09.245945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:09.245928 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:20:09.554394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:09.554312 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-4d3b7e" event={"ID":"9b9c7653-dc3d-46d5-814b-43fcbbf08496","Type":"ContainerStarted","Data":"25e0404f58f7fbbf80e1c6c587ebcb2882d00794003e79f242a6d038febc76f2"} Oct 30 03:20:11.567016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:11.566971 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-4d3b7e" event={"ID":"9b9c7653-dc3d-46d5-814b-43fcbbf08496","Type":"ContainerStarted","Data":"57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01"} Oct 30 03:20:11.605232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:11.605170 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7576-4d3b7e" podStartSLOduration=1.732877004 podStartE2EDuration="3.605154858s" totalImagesPullingTime="1.872277854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:11.603960086 +0000 UTC m=+1039.589079973" watchObservedRunningTime="2026-10-30 03:20:11.605154858 +0000 UTC m=+1039.590274746" Oct 30 03:20:11.749369 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:20:11.749331 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:20:12.517253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:12.517219 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-4d3b7e"] Oct 30 03:20:12.527605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:12.527579 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-4d3b7e"] Oct 30 03:20:13.184765 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.184733 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk"] Oct 30 03:20:13.185219 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.185194 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" containerName="test-curl-7576-4d3b7e" Oct 30 03:20:13.185219 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.185205 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" containerName="test-curl-7576-4d3b7e" Oct 30 03:20:13.185294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.185285 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" containerName="test-curl-7576-4d3b7e" Oct 30 03:20:13.188667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.188648 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.203679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.203655 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:20:13.216371 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.216331 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:20:13.216371 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.216345 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:20:13.216631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.216530 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:20:13.219314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.219289 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk"] Oct 30 03:20:13.298707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.298675 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.298892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.298734 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.298892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.298804 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2r4n\" (UniqueName: \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.400148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.400111 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.400333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.400169 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.400333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.400202 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2r4n\" (UniqueName: \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.400884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.400860 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.402600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.402576 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.420845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.420813 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2r4n\" (UniqueName: \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\") pod \"maas-api-revoke-keys-e2e-mt-b132360f-fhcbk\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.503308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.503275 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:13.578499 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.578404 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7576-4d3b7e" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" containerName="test-curl-7576-4d3b7e" containerID="cri-o://57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01" gracePeriod=2 Oct 30 03:20:13.592554 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:20:13.592507 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7576-4d3b7e\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:13.808709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:13.808678 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk"] Oct 30 03:20:13.809786 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:20:13.809757 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29053d0c_e919_469c_af40_00395ddc3a31.slice/crio-2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673 WatchSource:0}: Error finding container 2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673: Status 404 returned error can't find the container with id 2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673 Oct 30 03:20:14.585009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:14.584974 4560 generic.go:356] "Generic (PLEG): container finished" podID="29053d0c-e919-469c-af40-00395ddc3a31" containerID="30f9beadedb28a2d69e597a5d56b92163c7f3d2484931b76dde6f0a73dda6c1b" exitCode=0 Oct 30 03:20:14.585359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:14.585057 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" event={"ID":"29053d0c-e919-469c-af40-00395ddc3a31","Type":"ContainerDied","Data":"30f9beadedb28a2d69e597a5d56b92163c7f3d2484931b76dde6f0a73dda6c1b"} Oct 30 03:20:14.585359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:14.585116 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" event={"ID":"29053d0c-e919-469c-af40-00395ddc3a31","Type":"ContainerStarted","Data":"2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673"} Oct 30 03:20:15.745597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.745570 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:15.776277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.776251 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:15.832655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.826817 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\" (UniqueName: \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\") pod \"29053d0c-e919-469c-af40-00395ddc3a31\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " Oct 30 03:20:15.832655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.827381 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\" (UniqueName: \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\") pod \"9b9c7653-dc3d-46d5-814b-43fcbbf08496\" (UID: \"9b9c7653-dc3d-46d5-814b-43fcbbf08496\") " Oct 30 03:20:15.832655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.827509 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\") pod \"29053d0c-e919-469c-af40-00395ddc3a31\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " Oct 30 03:20:15.832655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.827615 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\") pod \"29053d0c-e919-469c-af40-00395ddc3a31\" (UID: \"29053d0c-e919-469c-af40-00395ddc3a31\") " Oct 30 03:20:15.832655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.831891 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca" pod "29053d0c-e919-469c-af40-00395ddc3a31" (UID: "29053d0c-e919-469c-af40-00395ddc3a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:15.838128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.837392 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca" pod "29053d0c-e919-469c-af40-00395ddc3a31" (UID: "29053d0c-e919-469c-af40-00395ddc3a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:15.838907 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.838868 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr" pod "9b9c7653-dc3d-46d5-814b-43fcbbf08496" (UID: "9b9c7653-dc3d-46d5-814b-43fcbbf08496"). InnerVolumeSpecName "kube-api-access-llhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:15.851426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.851369 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n" pod "29053d0c-e919-469c-af40-00395ddc3a31" (UID: "29053d0c-e919-469c-af40-00395ddc3a31"). InnerVolumeSpecName "kube-api-access-f2r4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:15.949115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.949075 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhtr\" (UniqueName: \"kubernetes.io/projected/9b9c7653-dc3d-46d5-814b-43fcbbf08496-kube-api-access-llhtr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:15.949115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.949109 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:15.949115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.949123 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29053d0c-e919-469c-af40-00395ddc3a31-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:15.949356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:15.949135 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2r4n\" (UniqueName: \"kubernetes.io/projected/29053d0c-e919-469c-af40-00395ddc3a31-kube-api-access-f2r4n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:16.152934 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.152855 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" path="/var/lib/kubelet/pods/9b9c7653-dc3d-46d5-814b-43fcbbf08496/volumes" Oct 30 03:20:16.601863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.601832 4560 generic.go:356] "Generic (PLEG): container finished" podID="9b9c7653-dc3d-46d5-814b-43fcbbf08496" containerID="57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01" exitCode=137 Oct 30 03:20:16.602007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.601891 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-4d3b7e" Oct 30 03:20:16.602007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.601935 4560 scope.go:122] "RemoveContainer" containerID="57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01" Oct 30 03:20:16.603481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.603460 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" Oct 30 03:20:16.603640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.603463 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk" event={"ID":"29053d0c-e919-469c-af40-00395ddc3a31","Type":"ContainerDied","Data":"2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673"} Oct 30 03:20:16.603640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.603604 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dbf2a1349decd19a252fc33ddd61c871c3c3b9060fc7d56f486aa8575949673" Oct 30 03:20:16.614796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.614772 4560 scope.go:122] "RemoveContainer" containerID="57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01" Oct 30 03:20:16.615110 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:20:16.615089 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01\": container with ID starting with 57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01 not found: ID does not exist" containerID="57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01" Oct 30 03:20:16.615162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:16.615116 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01"} err="failed to get container status \"57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01\": rpc error: code = NotFound desc = could not find container \"57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01\": container with ID starting with 57aa3eaf49cd7344fab6c7839f2d330bf3cbe9710aff586349004d96998ebf01 not found: ID does not exist" Oct 30 03:20:28.673769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.673680 4560 generic.go:356] "Generic (PLEG): container finished" podID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerID="5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805" exitCode=0 Oct 30 03:20:28.673769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.673755 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerDied","Data":"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805"} Oct 30 03:20:28.674206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.674153 4560 scope.go:122] "RemoveContainer" containerID="5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805" Oct 30 03:20:28.675588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.675561 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:28.675715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.675643 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:28.675715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:28.675691 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:29.175540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.173840 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7"] Oct 30 03:20:29.175540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.174652 4560 cpu_manager.go:414] "containerMap: removing container" podUID="29053d0c-e919-469c-af40-00395ddc3a31" containerName="revoke-keys" Oct 30 03:20:29.175540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.174671 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29053d0c-e919-469c-af40-00395ddc3a31" containerName="revoke-keys" Oct 30 03:20:29.175540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.174835 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="29053d0c-e919-469c-af40-00395ddc3a31" containerName="revoke-keys" Oct 30 03:20:29.178929 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.178903 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.194722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.194693 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:20:29.194876 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.194781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:20:29.194876 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.194860 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:20:29.194998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.194885 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:20:29.196999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.196974 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7"] Oct 30 03:20:29.302935 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.302902 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88946\" (UniqueName: \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.303139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.303089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.303210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.303154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.407540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.404392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.407540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.404489 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.407540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.404621 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88946\" (UniqueName: \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.407540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.405924 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.412057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.409691 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.424883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.424857 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88946\" (UniqueName: \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\") pod \"maas-api-revoke-sub-a23a9cc7-m26v7\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.494199 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.494169 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:29.686884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.686842 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerStarted","Data":"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a"} Oct 30 03:20:29.811840 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:20:29.811801 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e987b2_65e9_4b55_a91c_79c30fb4ded1.slice/crio-1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d WatchSource:0}: Error finding container 1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d: Status 404 returned error can't find the container with id 1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d Oct 30 03:20:29.812251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:29.812224 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7"] Oct 30 03:20:30.103464 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.103367 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:20:30.104602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.104578 4560 patch_prober.go:29] interesting pod/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 03:20:30.104720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.104642 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 03:20:30.696876 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.696839 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" event={"ID":"32e987b2-65e9-4b55-a91c-79c30fb4ded1","Type":"ContainerStarted","Data":"a98b6c6d892a49ae5a36eee475314803af5d9f81dbb6042b5e21652dfd373897"} Oct 30 03:20:30.697354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.696903 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" event={"ID":"32e987b2-65e9-4b55-a91c-79c30fb4ded1","Type":"ContainerStarted","Data":"1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d"} Oct 30 03:20:30.738318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:30.738261 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" podStartSLOduration=1.738247167 podStartE2EDuration="1.738247167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:30.73775464 +0000 UTC m=+1058.722874542" watchObservedRunningTime="2026-10-30 03:20:30.738247167 +0000 UTC m=+1058.723367055" Oct 30 03:20:31.103341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:31.103304 4560 patch_prober.go:29] interesting pod/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 03:20:31.103585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:31.103353 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 03:20:31.713254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:31.713221 4560 generic.go:356] "Generic (PLEG): container finished" podID="32e987b2-65e9-4b55-a91c-79c30fb4ded1" containerID="a98b6c6d892a49ae5a36eee475314803af5d9f81dbb6042b5e21652dfd373897" exitCode=0 Oct 30 03:20:31.713712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:31.713310 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" event={"ID":"32e987b2-65e9-4b55-a91c-79c30fb4ded1","Type":"ContainerDied","Data":"a98b6c6d892a49ae5a36eee475314803af5d9f81dbb6042b5e21652dfd373897"} Oct 30 03:20:32.103306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.103225 4560 patch_prober.go:29] interesting pod/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 03:20:32.103306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.103273 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 03:20:32.901710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.901685 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:32.946135 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.946107 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\" (UniqueName: \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\") pod \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " Oct 30 03:20:32.946301 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.946165 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\") pod \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " Oct 30 03:20:32.946301 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.946291 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\") pod \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\" (UID: \"32e987b2-65e9-4b55-a91c-79c30fb4ded1\") " Oct 30 03:20:32.946646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.946616 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca" pod "32e987b2-65e9-4b55-a91c-79c30fb4ded1" (UID: "32e987b2-65e9-4b55-a91c-79c30fb4ded1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:32.946802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.946784 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:32.948243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.948217 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946" pod "32e987b2-65e9-4b55-a91c-79c30fb4ded1" (UID: "32e987b2-65e9-4b55-a91c-79c30fb4ded1"). InnerVolumeSpecName "kube-api-access-88946". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:32.948360 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:32.948277 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca" pod "32e987b2-65e9-4b55-a91c-79c30fb4ded1" (UID: "32e987b2-65e9-4b55-a91c-79c30fb4ded1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:33.048173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.048136 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e987b2-65e9-4b55-a91c-79c30fb4ded1-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:33.048173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.048169 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88946\" (UniqueName: \"kubernetes.io/projected/32e987b2-65e9-4b55-a91c-79c30fb4ded1-kube-api-access-88946\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:20:33.104206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.104169 4560 patch_prober.go:29] interesting pod/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 03:20:33.104384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.104224 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 03:20:33.725933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.725901 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" Oct 30 03:20:33.726103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.725928 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7" event={"ID":"32e987b2-65e9-4b55-a91c-79c30fb4ded1","Type":"ContainerDied","Data":"1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d"} Oct 30 03:20:33.726103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:33.725978 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a69dc2302ef5f00bd86844853e10b06815a837385c83162f5310be0c29b2b7d" Oct 30 03:20:34.103759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:34.103670 4560 patch_prober.go:29] interesting pod/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 03:20:34.103759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:34.103729 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 03:20:34.437980 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:34.437898 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:20:35.109366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:35.109333 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:20:35.110662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:35.110639 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:20:41.748682 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:20:41.748588 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:20:44.817103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.817017 4560 generic.go:356] "Generic (PLEG): container finished" podID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerID="ce66890336a2b600b9d5e0e976ca7d2f1da779b013ab8b4882401ccc082c6a04" exitCode=0 Oct 30 03:20:44.817535 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.817094 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerDied","Data":"ce66890336a2b600b9d5e0e976ca7d2f1da779b013ab8b4882401ccc082c6a04"} Oct 30 03:20:44.817535 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.817484 4560 scope.go:122] "RemoveContainer" containerID="ce66890336a2b600b9d5e0e976ca7d2f1da779b013ab8b4882401ccc082c6a04" Oct 30 03:20:44.818939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.818914 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:44.819009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.818983 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:44.819046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:44.819017 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:20:45.505922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:45.505893 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:45.824082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:45.824005 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerStarted","Data":"a4d774efef7ce87099df2a0300df941305812f3ec7a0b233579ba82a06fbbe15"} Oct 30 03:20:46.089677 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:46.089595 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:46.090941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:46.090915 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:46.091049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:46.090973 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:47.090278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:47.090244 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:47.090733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:47.090309 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:48.090076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:48.090039 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:48.090284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:48.090100 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:49.090020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:49.089978 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:49.090439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:49.090036 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:50.090162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:50.090116 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:50.090726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:50.090185 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:50.285043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:50.284992 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk"] Oct 30 03:20:50.288853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:50.288816 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b132360f-fhcbk"] Oct 30 03:20:51.100420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:51.099961 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 03:20:51.100420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:51.100038 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 03:20:52.097922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:52.097887 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:52.098371 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:52.098301 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:52.099096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:52.099074 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:20:52.154999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:52.154967 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29053d0c-e919-469c-af40-00395ddc3a31" path="/var/lib/kubelet/pods/29053d0c-e919-469c-af40-00395ddc3a31/volumes" Oct 30 03:20:52.370320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:20:52.370236 4560 scope.go:122] "RemoveContainer" containerID="f44ddaf0fdebe64eb73e8f25c25636b4a542d09d95e87a2145cea5d745d5693b" Oct 30 03:21:10.127768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:10.127734 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:21:10.128460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:10.127997 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" containerID="cri-o://aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a" gracePeriod=30 Oct 30 03:21:11.749663 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:11.749625 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:21:15.393498 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.393436 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:21:15.555602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555564 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.555882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555655 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.555882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555680 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.555882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555710 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.555882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555727 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.555882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555922 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555951 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.555996 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\") pod \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\" (UID: \"d16ec8c7-df43-4496-a6f2-3f6f83f93768\") " Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556035 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556078 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556089 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:15.556128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556113 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:15.556463 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556146 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:15.556541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556481 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.556541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556502 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.556719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556559 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.556719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556574 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.556719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.556587 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.558078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.558039 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:15.558078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.558063 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:21:15.558078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.558074 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:15.558286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.558209 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn" pod "d16ec8c7-df43-4496-a6f2-3f6f83f93768" (UID: "d16ec8c7-df43-4496-a6f2-3f6f83f93768"). InnerVolumeSpecName "kube-api-access-lfpmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:15.658276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.658188 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.658276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.658221 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.658276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.658234 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpmn\" (UniqueName: \"kubernetes.io/projected/d16ec8c7-df43-4496-a6f2-3f6f83f93768-kube-api-access-lfpmn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:15.658276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:15.658243 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16ec8c7-df43-4496-a6f2-3f6f83f93768-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:16.013429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.013395 4560 generic.go:356] "Generic (PLEG): container finished" podID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerID="aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a" exitCode=0 Oct 30 03:21:16.013635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.013443 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" Oct 30 03:21:16.013635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.013442 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerDied","Data":"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a"} Oct 30 03:21:16.013635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.013501 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w" event={"ID":"d16ec8c7-df43-4496-a6f2-3f6f83f93768","Type":"ContainerDied","Data":"4c2f42887676af242c015907356aa18a45f52995ec0f1bfbc877c700f4c66aa0"} Oct 30 03:21:16.013635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.013546 4560 scope.go:122] "RemoveContainer" containerID="aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a" Oct 30 03:21:16.032959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.032938 4560 scope.go:122] "RemoveContainer" containerID="5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805" Oct 30 03:21:16.041418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.041398 4560 scope.go:122] "RemoveContainer" containerID="aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a" Oct 30 03:21:16.041696 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:16.041673 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a\": container with ID starting with aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a not found: ID does not exist" containerID="aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a" Oct 30 03:21:16.041787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.041707 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a"} err="failed to get container status \"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a\": rpc error: code = NotFound desc = could not find container \"aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a\": container with ID starting with aea2d33e4c104beda5e711f257e9ff1fb8966ff946797fbc314eadadf701a25a not found: ID does not exist" Oct 30 03:21:16.041787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.041732 4560 scope.go:122] "RemoveContainer" containerID="5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805" Oct 30 03:21:16.042022 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:16.042003 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805\": container with ID starting with 5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805 not found: ID does not exist" containerID="5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805" Oct 30 03:21:16.042056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.042028 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805"} err="failed to get container status \"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805\": rpc error: code = NotFound desc = could not find container \"5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805\": container with ID starting with 5473316d9bf82a613a7be947e255d2aa9f402d409b6ec5ee6e363559038da805 not found: ID does not exist" Oct 30 03:21:16.079567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.079533 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:21:16.092146 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.092114 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3e4daf-openshift-default-7d5fddbfc-sc46w"] Oct 30 03:21:16.153276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:16.153243 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" path="/var/lib/kubelet/pods/d16ec8c7-df43-4496-a6f2-3f6f83f93768/volumes" Oct 30 03:21:19.039574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.039509 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:21:19.040235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040206 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:19.040235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040230 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040249 4560 cpu_manager.go:414] "containerMap: removing container" podUID="32e987b2-65e9-4b55-a91c-79c30fb4ded1" containerName="revoke-keys" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040284 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e987b2-65e9-4b55-a91c-79c30fb4ded1" containerName="revoke-keys" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040316 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040324 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040496 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:19.040857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.040508 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e987b2-65e9-4b55-a91c-79c30fb4ded1" containerName="revoke-keys" Oct 30 03:21:19.045531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.045500 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.067243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.067208 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:21:19.095374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095339 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8lhf\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095388 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095472 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095603 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095696 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095730 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095773 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.095973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.095940 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203094 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203207 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203315 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203425 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203491 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203557 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8lhf\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203623 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203687 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203775 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.203995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.203966 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.204384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.204338 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.205125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.205097 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.206653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.205892 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.206653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.206044 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.208279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.208248 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.208784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.208761 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.225116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.225090 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.225223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.225185 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8lhf\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9lv85\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.361204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.361114 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:19.680849 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.680821 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:21:19.682226 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:21:19.682197 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8031d3d_6736_42c6_8037_39550d24fae7.slice/crio-fb77394f79b6a5e8ad562aa3c4e049c465e5c25997d296265cd3560462a8b9b6 WatchSource:0}: Error finding container fb77394f79b6a5e8ad562aa3c4e049c465e5c25997d296265cd3560462a8b9b6: Status 404 returned error can't find the container with id fb77394f79b6a5e8ad562aa3c4e049c465e5c25997d296265cd3560462a8b9b6 Oct 30 03:21:19.684661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.684629 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:21:19.684743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.684694 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:21:19.684822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:19.684749 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:21:20.040486 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.040452 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" event={"ID":"b8031d3d-6736-42c6-8037-39550d24fae7","Type":"ContainerStarted","Data":"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0"} Oct 30 03:21:20.041020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.040506 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" event={"ID":"b8031d3d-6736-42c6-8037-39550d24fae7","Type":"ContainerStarted","Data":"fb77394f79b6a5e8ad562aa3c4e049c465e5c25997d296265cd3560462a8b9b6"} Oct 30 03:21:20.362425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.362316 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:20.364204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.364172 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:20.364344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.364237 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:20.902553 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.901329 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podStartSLOduration=1.901314173 podStartE2EDuration="1.901314173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:20.110308786 +0000 UTC m=+1108.095428670" watchObservedRunningTime="2026-10-30 03:21:20.901314173 +0000 UTC m=+1108.886434060" Oct 30 03:21:20.903491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.903463 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:21:20.903772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:20.903747 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" containerID="cri-o://a4d774efef7ce87099df2a0300df941305812f3ec7a0b233579ba82a06fbbe15" gracePeriod=30 Oct 30 03:21:21.367542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:21.365951 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:21.367542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:21.366034 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:22.100774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:22.100738 4560 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:21:22.100915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:22.100792 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:21:22.372011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:22.371920 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:22.372663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:22.372624 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:23.369542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:23.368902 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:23.369542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:23.368964 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:24.370543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:24.368873 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:24.370543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:24.368934 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:25.366930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:25.366890 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:25.367214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:25.367184 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:26.108078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.105927 4560 generic.go:356] "Generic (PLEG): container finished" podID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerID="a4d774efef7ce87099df2a0300df941305812f3ec7a0b233579ba82a06fbbe15" exitCode=0 Oct 30 03:21:26.108078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.106000 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerDied","Data":"a4d774efef7ce87099df2a0300df941305812f3ec7a0b233579ba82a06fbbe15"} Oct 30 03:21:26.108078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.106082 4560 scope.go:122] "RemoveContainer" containerID="ce66890336a2b600b9d5e0e976ca7d2f1da779b013ab8b4882401ccc082c6a04" Oct 30 03:21:26.181183 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.180110 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:21:26.305836 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.305794 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.305925 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306052 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306085 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306127 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306178 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306231 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306268 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306297 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\") pod \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\" (UID: \"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b\") " Oct 30 03:21:26.306745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306484 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.306745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306674 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.306745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306710 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.307665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.306975 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:26.307665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.307328 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.307665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.307352 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.307665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.307364 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.307665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.307378 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.308008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.307978 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.309220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.309189 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "kube-api-access-gw7v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:26.309343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.309215 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:26.309343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.309298 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:21:26.309343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.309313 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy" pod "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" (UID: "f5e01cfa-3e66-4646-8a24-4d6dedb6b70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.361634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.361594 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:26.361823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.361653 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:26.408595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.408562 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.408595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.408588 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.408595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.408598 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7v9\" (UniqueName: \"kubernetes.io/projected/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-kube-api-access-gw7v9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.408807 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.408607 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.408807 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:26.408615 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:27.114829 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.114788 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" event={"ID":"f5e01cfa-3e66-4646-8a24-4d6dedb6b70b","Type":"ContainerDied","Data":"5d643e649101d7d9b87c02658565f1be6c9a6d470cbb4a102c9ef40957cdd692"} Oct 30 03:21:27.115342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.114799 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65" Oct 30 03:21:27.115342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.114863 4560 scope.go:122] "RemoveContainer" containerID="a4d774efef7ce87099df2a0300df941305812f3ec7a0b233579ba82a06fbbe15" Oct 30 03:21:27.180991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.180957 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:21:27.195239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.195200 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-slz65"] Oct 30 03:21:27.361704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.361669 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 03:21:27.361869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:27.361724 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 03:21:28.154230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:28.154190 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" path="/var/lib/kubelet/pods/f5e01cfa-3e66-4646-8a24-4d6dedb6b70b/volumes" Oct 30 03:21:28.527053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:28.527016 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:28.527459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:28.527425 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:28.528723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:28.528701 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:21:31.179207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179172 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-aea268"] Oct 30 03:21:31.179698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179683 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.179748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179699 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.179748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179731 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.179748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179736 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.179838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179806 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16ec8c7-df43-4496-a6f2-3f6f83f93768" containerName="istio-proxy" Oct 30 03:21:31.179838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179817 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.179838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.179825 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e01cfa-3e66-4646-8a24-4d6dedb6b70b" containerName="istio-proxy" Oct 30 03:21:31.184769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.184752 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:31.198644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.198617 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-aea268"] Oct 30 03:21:31.208391 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.208365 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 03:21:31.264308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.264272 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmn6l\" (UniqueName: \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\") pod \"test-curl-7576-aea268\" (UID: \"54d85c03-1365-44af-a506-e2a02a0b893c\") " pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:31.365709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.365675 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmn6l\" (UniqueName: \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\") pod \"test-curl-7576-aea268\" (UID: \"54d85c03-1365-44af-a506-e2a02a0b893c\") " pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:31.385889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.385857 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmn6l\" (UniqueName: \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\") pod \"test-curl-7576-aea268\" (UID: \"54d85c03-1365-44af-a506-e2a02a0b893c\") " pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:31.498037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.498005 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:31.797124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:31.797099 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-aea268"] Oct 30 03:21:31.798485 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:21:31.798464 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54d85c03_1365_44af_a506_e2a02a0b893c.slice/crio-b394b898e4314c9fd5c7c8baab757d29d420b1836a471b0f64f42df876c4abee WatchSource:0}: Error finding container b394b898e4314c9fd5c7c8baab757d29d420b1836a471b0f64f42df876c4abee: Status 404 returned error can't find the container with id b394b898e4314c9fd5c7c8baab757d29d420b1836a471b0f64f42df876c4abee Oct 30 03:21:32.152362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:32.152286 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-aea268" event={"ID":"54d85c03-1365-44af-a506-e2a02a0b893c","Type":"ContainerStarted","Data":"e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796"} Oct 30 03:21:32.152362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:32.152328 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-aea268" event={"ID":"54d85c03-1365-44af-a506-e2a02a0b893c","Type":"ContainerStarted","Data":"b394b898e4314c9fd5c7c8baab757d29d420b1836a471b0f64f42df876c4abee"} Oct 30 03:21:32.187705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:32.187658 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7576-aea268" podStartSLOduration=1.187644851 podStartE2EDuration="1.187644851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:32.184871696 +0000 UTC m=+1120.169991584" watchObservedRunningTime="2026-10-30 03:21:32.187644851 +0000 UTC m=+1120.172764739" Oct 30 03:21:33.051385 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:33.051350 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-aea268"] Oct 30 03:21:33.053924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:33.053897 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-aea268"] Oct 30 03:21:34.158669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:34.158582 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7576-aea268" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" containerName="test-curl-7576-aea268" containerID="cri-o://e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796" gracePeriod=2 Oct 30 03:21:36.332914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:36.332877 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:36.426550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:36.426447 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\" (UniqueName: \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\") pod \"54d85c03-1365-44af-a506-e2a02a0b893c\" (UID: \"54d85c03-1365-44af-a506-e2a02a0b893c\") " Oct 30 03:21:36.428581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:36.428540 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l" pod "54d85c03-1365-44af-a506-e2a02a0b893c" (UID: "54d85c03-1365-44af-a506-e2a02a0b893c"). InnerVolumeSpecName "kube-api-access-vmn6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:36.528001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:36.527957 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmn6l\" (UniqueName: \"kubernetes.io/projected/54d85c03-1365-44af-a506-e2a02a0b893c-kube-api-access-vmn6l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:37.192845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:37.187502 4560 generic.go:356] "Generic (PLEG): container finished" podID="54d85c03-1365-44af-a506-e2a02a0b893c" containerID="e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796" exitCode=137 Oct 30 03:21:37.192845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:37.187626 4560 scope.go:122] "RemoveContainer" containerID="e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796" Oct 30 03:21:37.192845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:37.187798 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:37.208473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:37.208445 4560 scope.go:122] "RemoveContainer" containerID="e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796" Oct 30 03:21:37.208866 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:37.208843 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796\": container with ID starting with e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796 not found: ID does not exist" containerID="e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796" Oct 30 03:21:37.208961 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:37.208878 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796"} err="failed to get container status \"e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796\": rpc error: code = NotFound desc = could not find container \"e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796\": container with ID starting with e0df0e91f09ecd11f9390719ac77fe33bd12c088fa88f637f89a87cb4f844796 not found: ID does not exist" Oct 30 03:21:37.216450 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:37.216419 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7576-aea268\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" pod="opendatahub/test-curl-7576-aea268" Oct 30 03:21:38.160125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:38.160087 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" path="/var/lib/kubelet/pods/54d85c03-1365-44af-a506-e2a02a0b893c/volumes" Oct 30 03:21:38.574552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:38.574503 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:21:41.749285 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:41.749238 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:21:53.638760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.638701 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" podUID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" containerName="main" containerID="cri-o://c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a" gracePeriod=45 Oct 30 03:21:53.898640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.898572 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:21:53.945631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.945592 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\" (UniqueName: \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.945811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.945648 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.945811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.945756 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.945811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.945782 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.945976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.945889 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.946033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946001 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\") pod \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\" (UID: \"c27db272-504d-4191-aeb2-2bc7b7d3b96a\") " Oct 30 03:21:53.946954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946228 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:53.946954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946361 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:53.946954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946866 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:53.946954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946751 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:53.946954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.946932 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:53.948560 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.948487 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "kube-api-access-tjgt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:53.949633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.949600 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:53.949742 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:53.949671 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm" pod "c27db272-504d-4191-aeb2-2bc7b7d3b96a" (UID: "c27db272-504d-4191-aeb2-2bc7b7d3b96a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:54.048222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.048190 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:54.048222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.048220 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgt8\" (UniqueName: \"kubernetes.io/projected/c27db272-504d-4191-aeb2-2bc7b7d3b96a-kube-api-access-tjgt8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:54.048416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.048230 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27db272-504d-4191-aeb2-2bc7b7d3b96a-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:54.048416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.048243 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27db272-504d-4191-aeb2-2bc7b7d3b96a-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:21:54.318268 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.318233 4560 generic.go:356] "Generic (PLEG): container finished" podID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" containerID="c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a" exitCode=0 Oct 30 03:21:54.318450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.318292 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" Oct 30 03:21:54.318450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.318311 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" event={"ID":"c27db272-504d-4191-aeb2-2bc7b7d3b96a","Type":"ContainerDied","Data":"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a"} Oct 30 03:21:54.318450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.318357 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9" event={"ID":"c27db272-504d-4191-aeb2-2bc7b7d3b96a","Type":"ContainerDied","Data":"89b39e498d29e755571242828b7a94a6a69f6ae82729bebddcdae6c65f25e5ce"} Oct 30 03:21:54.318450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.318378 4560 scope.go:122] "RemoveContainer" containerID="c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a" Oct 30 03:21:54.328718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.328694 4560 scope.go:122] "RemoveContainer" containerID="c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a" Oct 30 03:21:54.329009 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:21:54.328987 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a\": container with ID starting with c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a not found: ID does not exist" containerID="c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a" Oct 30 03:21:54.329090 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.329016 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a"} err="failed to get container status \"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a\": rpc error: code = NotFound desc = could not find container \"c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a\": container with ID starting with c533736ecca578f4ca7122f51a079d6a00d918a1e531ec58a201550fb213914a not found: ID does not exist" Oct 30 03:21:54.372023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.371988 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:21:54.377306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:54.377280 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b132360f/e2e-lifecycle-model-b132360f-kserve-649fc89949-85kw9"] Oct 30 03:21:56.152550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:21:56.152495 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" path="/var/lib/kubelet/pods/c27db272-504d-4191-aeb2-2bc7b7d3b96a/volumes" Oct 30 03:22:11.749400 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:22:11.749363 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:22:41.749555 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:22:41.749491 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:22:44.069542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:44.058634 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:22:44.069542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:44.058719 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:22:44.534878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:44.534838 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9lv85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:22:44.535187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:44.535138 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:22:56.865068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.865026 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:22:56.867150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.867117 4560 cpu_manager.go:414] "containerMap: removing container" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" containerName="test-curl-7576-aea268" Oct 30 03:22:56.867328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.867314 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" containerName="test-curl-7576-aea268" Oct 30 03:22:56.867484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.867472 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" containerName="main" Oct 30 03:22:56.867609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.867595 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" containerName="main" Oct 30 03:22:56.867969 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.867949 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d85c03-1365-44af-a506-e2a02a0b893c" containerName="test-curl-7576-aea268" Oct 30 03:22:56.868097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.868086 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27db272-504d-4191-aeb2-2bc7b7d3b96a" containerName="main" Oct 30 03:22:56.879356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.879264 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.903761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.903729 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:22:56.907737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.907708 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6c106c-openshift-default-dockercfg-k7lsk\"" Oct 30 03:22:56.968177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968147 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968225 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8xz\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968356 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968466 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968405 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968466 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968429 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968500 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968595 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968645 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:56.968797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:56.968728 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070237 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070263 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070316 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070426 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8xz\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070474 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070573 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070599 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070626 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070655 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070738 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.070944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.070811 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.071323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.071301 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.071440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.071381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.071960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.071937 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.074187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.074162 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.074587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.074510 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.092612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.092543 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.098311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.098192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8xz\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.215842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.215811 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:57.539696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.539596 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:22:57.542751 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:22:57.542721 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c78512_53c3_4c4a_905d_147565cda2c1.slice/crio-098ae42188e96d75a4c1c47f933e5fcbaa7d02cfd312ee896b23a88c2cb15b06 WatchSource:0}: Error finding container 098ae42188e96d75a4c1c47f933e5fcbaa7d02cfd312ee896b23a88c2cb15b06: Status 404 returned error can't find the container with id 098ae42188e96d75a4c1c47f933e5fcbaa7d02cfd312ee896b23a88c2cb15b06 Oct 30 03:22:57.545805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.545766 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:22:57.545912 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.545861 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:22:57.545912 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.545906 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:22:57.712306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.712093 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerStarted","Data":"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78"} Oct 30 03:22:57.712306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.712160 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerStarted","Data":"098ae42188e96d75a4c1c47f933e5fcbaa7d02cfd312ee896b23a88c2cb15b06"} Oct 30 03:22:57.763151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:57.763080 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podStartSLOduration=1.7630618089999999 podStartE2EDuration="1.763061809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:57.761702461 +0000 UTC m=+1205.746822378" watchObservedRunningTime="2026-10-30 03:22:57.763061809 +0000 UTC m=+1205.748181698" Oct 30 03:22:58.216281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:58.216250 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:58.221429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:58.221404 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:58.717764 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:58.717738 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:22:58.718690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:22:58.718668 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:23:06.400372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.399398 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:23:06.404617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.404591 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.430072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.430031 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:23:06.471667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.471631 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e0f003-openshift-default-dockercfg-6vz5t\"" Oct 30 03:23:06.484604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484567 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.484777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484619 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.484777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484711 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.484777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484755 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.484944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484844 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.484944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484885 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.485046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484942 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.485046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.484987 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.485046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.485033 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8lf\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.586736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586679 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.586736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586752 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586782 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586828 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586860 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8lf\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586946 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.586990 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587026 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587055 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587180 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587219 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587694 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587795 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.587924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.587881 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.589826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.589802 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.590018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.590000 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.607443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.607401 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.608024 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.608003 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8lf\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\") pod \"e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:06.726833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:06.726791 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:07.062212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.062183 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:23:07.063297 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:23:07.063268 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c7f463_e0d5_4db5_860e_390bdd9d018a.slice/crio-ff607cdc6ced7b2b5b203afe79a9fc92aeae27bd282604d2116ba47cf18ea60c WatchSource:0}: Error finding container ff607cdc6ced7b2b5b203afe79a9fc92aeae27bd282604d2116ba47cf18ea60c: Status 404 returned error can't find the container with id ff607cdc6ced7b2b5b203afe79a9fc92aeae27bd282604d2116ba47cf18ea60c Oct 30 03:23:07.066007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.065965 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:07.066086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.066056 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:07.066124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.066098 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:07.770967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.770927 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerStarted","Data":"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6"} Oct 30 03:23:07.771388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.770978 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerStarted","Data":"ff607cdc6ced7b2b5b203afe79a9fc92aeae27bd282604d2116ba47cf18ea60c"} Oct 30 03:23:07.832151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:07.832095 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podStartSLOduration=1.832080775 podStartE2EDuration="1.832080775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:07.831001973 +0000 UTC m=+1215.816121861" watchObservedRunningTime="2026-10-30 03:23:07.832080775 +0000 UTC m=+1215.817200727" Oct 30 03:23:08.735138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:08.735105 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:08.740732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:08.740702 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:08.775995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:08.775967 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:08.776913 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:08.776894 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:11.749266 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:23:11.749231 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:23:12.107489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.107397 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:23:12.111267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.111242 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.131444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.131409 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:23:12.185167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.185135 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0f003-dockercfg-nc729\"" Oct 30 03:23:12.195019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.192572 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0f003\"" Oct 30 03:23:12.289392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.289353 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.289669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.289645 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bhm\" (UniqueName: \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.390418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.390338 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.390609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.390458 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bhm\" (UniqueName: \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.391098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.391078 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.411508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.411473 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bhm\" (UniqueName: \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.425184 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.425157 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:12.757390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.757362 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:23:12.758855 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:23:12.758831 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a99879_045a_4f9e_ae52_368320149a9f.slice/crio-02c8712c8898ebaa2a275f2aa80dbefc529b0fad650bff00f47d35cbc060d325 WatchSource:0}: Error finding container 02c8712c8898ebaa2a275f2aa80dbefc529b0fad650bff00f47d35cbc060d325: Status 404 returned error can't find the container with id 02c8712c8898ebaa2a275f2aa80dbefc529b0fad650bff00f47d35cbc060d325 Oct 30 03:23:12.797826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:12.797792 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" event={"ID":"88a99879-045a-4f9e-ae52-368320149a9f","Type":"ContainerStarted","Data":"02c8712c8898ebaa2a275f2aa80dbefc529b0fad650bff00f47d35cbc060d325"} Oct 30 03:23:15.822641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:15.822603 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" event={"ID":"88a99879-045a-4f9e-ae52-368320149a9f","Type":"ContainerStarted","Data":"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04"} Oct 30 03:23:15.823167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:15.822684 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:15.867781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:15.867728 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" podStartSLOduration=1.809922918 podStartE2EDuration="3.867714185s" totalImagesPullingTime="2.057791267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:15.862715361 +0000 UTC m=+1223.847835262" watchObservedRunningTime="2026-10-30 03:23:15.867714185 +0000 UTC m=+1223.852834072" Oct 30 03:23:26.830905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:26.830876 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:23:41.017636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.017536 4560 generic.go:356] "Generic (PLEG): container finished" podID="35c78512-53c3-4c4a-905d-147565cda2c1" containerID="4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78" exitCode=0 Oct 30 03:23:41.017636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.017587 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerDied","Data":"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78"} Oct 30 03:23:41.018103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.017985 4560 scope.go:122] "RemoveContainer" containerID="4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78" Oct 30 03:23:41.019539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.019498 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:41.019621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.019581 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:41.019621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:41.019613 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:41.763540 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:23:41.759727 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:23:42.025826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:42.025745 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerStarted","Data":"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519"} Oct 30 03:23:42.216439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:42.216405 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:23:42.217765 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:42.217739 4560 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 03:23:42.217889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:42.217788 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 03:23:43.216905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:43.216864 4560 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 03:23:43.217319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:43.216925 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 03:23:43.717635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:43.717601 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:23:44.217173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:44.217137 4560 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 03:23:44.217600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:44.217204 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 03:23:45.216438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:45.216399 4560 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 03:23:45.216681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:45.216465 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 03:23:46.217153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:46.217123 4560 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 03:23:46.217643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:46.217173 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 03:23:47.437899 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:47.436788 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:23:47.439058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:47.439028 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:23:54.047431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.047383 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:23:54.070544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.064565 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.093781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.093747 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:23:54.094641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.094617 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b03a5e-openshift-default-dockercfg-xhd9h\"" Oct 30 03:23:54.133603 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133568 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133617 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133774 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133862 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmrk\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133890 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.133920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.133912 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.134071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.134053 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.134221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.134089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.243540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.235185 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.243826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.242212 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.243803 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmrk\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.244165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.244280 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.244565 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.244734 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.244915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.244902 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.245030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.245016 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.245166 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.245136 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.245692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.245670 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.246095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.246075 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.247302 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.247278 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.247696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.247312 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.247812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.247649 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.264580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.264548 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.275141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.275110 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.275458 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.275416 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmrk\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\") pod \"e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.414316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.414275 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:54.742759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.742722 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:23:54.743951 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:23:54.743927 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9c94d9_ebcf_4d7d_a1e4_399435057536.slice/crio-d5a93fd41acf285f5f8ca492ce1a72ac0faca5950093f2bdbbd169e3847573eb WatchSource:0}: Error finding container d5a93fd41acf285f5f8ca492ce1a72ac0faca5950093f2bdbbd169e3847573eb: Status 404 returned error can't find the container with id d5a93fd41acf285f5f8ca492ce1a72ac0faca5950093f2bdbbd169e3847573eb Oct 30 03:23:54.746673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.746640 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:54.746756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.746706 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:54.746756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:54.746733 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:55.116208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:55.116114 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerStarted","Data":"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062"} Oct 30 03:23:55.116208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:55.116195 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerStarted","Data":"d5a93fd41acf285f5f8ca492ce1a72ac0faca5950093f2bdbbd169e3847573eb"} Oct 30 03:23:55.415068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:55.414983 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:55.420846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:55.420818 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:55.466369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:55.466304 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podStartSLOduration=1.466287102 podStartE2EDuration="1.466287102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:55.162719843 +0000 UTC m=+1263.147839744" watchObservedRunningTime="2026-10-30 03:23:55.466287102 +0000 UTC m=+1263.451407010" Oct 30 03:23:56.122129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:56.122098 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:56.123164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:56.123140 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:23:58.134581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.134467 4560 generic.go:356] "Generic (PLEG): container finished" podID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerID="af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6" exitCode=0 Oct 30 03:23:58.134581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.134545 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerDied","Data":"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6"} Oct 30 03:23:58.135175 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.135156 4560 scope.go:122] "RemoveContainer" containerID="af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6" Oct 30 03:23:58.136630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.136608 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:58.136684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.136670 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:58.136722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:58.136700 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:23:59.141985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:59.141944 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerStarted","Data":"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa"} Oct 30 03:23:59.734576 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:59.734487 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:23:59.734767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:59.734693 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 03:23:59.734767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:23:59.734741 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 03:24:00.735784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:00.735747 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 03:24:00.736186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:00.735806 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 03:24:01.184482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:01.184387 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:24:01.743567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:01.736938 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 03:24:01.743567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:01.737001 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 03:24:02.735373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:02.735330 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 03:24:02.735573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:02.735405 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 03:24:03.735486 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:03.735443 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 03:24:03.736124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:03.736087 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 03:24:04.739672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:04.739638 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:24:04.740103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:04.739937 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:24:04.740562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:04.740540 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:24:06.185377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.185333 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" podUID="88a99879-045a-4f9e-ae52-368320149a9f" containerName="payload-pre-processing" containerID="cri-o://206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04" gracePeriod=25 Oct 30 03:24:06.445086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.445025 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:24:06.620002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.619966 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\" (UniqueName: \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\") pod \"88a99879-045a-4f9e-ae52-368320149a9f\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " Oct 30 03:24:06.620206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.620070 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\") pod \"88a99879-045a-4f9e-ae52-368320149a9f\" (UID: \"88a99879-045a-4f9e-ae52-368320149a9f\") " Oct 30 03:24:06.620473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.620449 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume" pod "88a99879-045a-4f9e-ae52-368320149a9f" (UID: "88a99879-045a-4f9e-ae52-368320149a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:24:06.622094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.622070 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm" pod "88a99879-045a-4f9e-ae52-368320149a9f" (UID: "88a99879-045a-4f9e-ae52-368320149a9f"). InnerVolumeSpecName "kube-api-access-v2bhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:24:06.721873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.721838 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a99879-045a-4f9e-ae52-368320149a9f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:24:06.721873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:06.721870 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bhm\" (UniqueName: \"kubernetes.io/projected/88a99879-045a-4f9e-ae52-368320149a9f-kube-api-access-v2bhm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:24:07.219686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.219651 4560 generic.go:356] "Generic (PLEG): container finished" podID="88a99879-045a-4f9e-ae52-368320149a9f" containerID="206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04" exitCode=0 Oct 30 03:24:07.220132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.219712 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" Oct 30 03:24:07.220132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.219735 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" event={"ID":"88a99879-045a-4f9e-ae52-368320149a9f","Type":"ContainerDied","Data":"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04"} Oct 30 03:24:07.220132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.219796 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn" event={"ID":"88a99879-045a-4f9e-ae52-368320149a9f","Type":"ContainerDied","Data":"02c8712c8898ebaa2a275f2aa80dbefc529b0fad650bff00f47d35cbc060d325"} Oct 30 03:24:07.220132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.219821 4560 scope.go:122] "RemoveContainer" containerID="206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04" Oct 30 03:24:07.230395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.230368 4560 scope.go:122] "RemoveContainer" containerID="206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04" Oct 30 03:24:07.230708 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:24:07.230682 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04\": container with ID starting with 206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04 not found: ID does not exist" containerID="206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04" Oct 30 03:24:07.230795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.230715 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04"} err="failed to get container status \"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04\": rpc error: code = NotFound desc = could not find container \"206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04\": container with ID starting with 206cfa61293ffebb7a03772f88ffdb72f218cac468a354757b47658b0721ef04 not found: ID does not exist" Oct 30 03:24:07.272059 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.272025 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:24:07.285054 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:07.285021 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-6544db76cf-47lbn"] Oct 30 03:24:08.154782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:08.154747 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a99879-045a-4f9e-ae52-368320149a9f" path="/var/lib/kubelet/pods/88a99879-045a-4f9e-ae52-368320149a9f/volumes" Oct 30 03:24:11.750164 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:24:11.750129 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:24:14.515392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.515356 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:24:14.515985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.515966 4560 cpu_manager.go:414] "containerMap: removing container" podUID="88a99879-045a-4f9e-ae52-368320149a9f" containerName="payload-pre-processing" Oct 30 03:24:14.516050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.515985 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a99879-045a-4f9e-ae52-368320149a9f" containerName="payload-pre-processing" Oct 30 03:24:14.516122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.516111 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a99879-045a-4f9e-ae52-368320149a9f" containerName="payload-pre-processing" Oct 30 03:24:14.522150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.522123 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.539920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.539896 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-dff241-openshift-default-dockercfg-j2mql\"" Oct 30 03:24:14.557036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.557000 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:24:14.608281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608245 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608297 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608399 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsp6\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608463 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608545 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608635 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608688 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608766 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.608843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.608808 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709338 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709382 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709418 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709460 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709503 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709552 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709720 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.709869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709767 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsp6\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.710013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709909 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.710013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.709968 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.710108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.710024 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.710163 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.710121 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.710307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.710282 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.712024 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.711995 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.712241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.712222 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.761822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.761790 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.770772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.768193 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsp6\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\") pod \"e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:14.844231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:14.844192 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:15.166890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.166863 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:24:15.168427 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:24:15.168394 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35751e1f_833d_490b_9464_fd65124c9197.slice/crio-f3a71a23e5087c62d9bbecd37c6a6b558f9e60e2e4e64c7280e25f7e8416b88c WatchSource:0}: Error finding container f3a71a23e5087c62d9bbecd37c6a6b558f9e60e2e4e64c7280e25f7e8416b88c: Status 404 returned error can't find the container with id f3a71a23e5087c62d9bbecd37c6a6b558f9e60e2e4e64c7280e25f7e8416b88c Oct 30 03:24:15.170953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.170926 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:15.171030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.170985 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:15.171030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.171016 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:15.267897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.267855 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerStarted","Data":"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f"} Oct 30 03:24:15.268095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.267928 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerStarted","Data":"f3a71a23e5087c62d9bbecd37c6a6b558f9e60e2e4e64c7280e25f7e8416b88c"} Oct 30 03:24:15.311904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.311663 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podStartSLOduration=1.311645178 podStartE2EDuration="1.311645178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:15.311297968 +0000 UTC m=+1283.296417859" watchObservedRunningTime="2026-10-30 03:24:15.311645178 +0000 UTC m=+1283.296765071" Oct 30 03:24:15.845116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:15.845082 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:16.850586 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:16.850554 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:17.279263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:17.279232 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:17.280295 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:17.280275 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:24:19.521698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.521666 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:24:19.525674 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.525655 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.541647 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.541624 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dff241\"" Oct 30 03:24:19.554414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.554386 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dff241-dockercfg-zn7jq\"" Oct 30 03:24:19.557323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.557295 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:24:19.572108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.572076 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.572238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.572132 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6dl\" (UniqueName: \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.677924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.677880 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.678127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.677945 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6dl\" (UniqueName: \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.680896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.678957 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.697426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.697397 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6dl\" (UniqueName: \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\") pod \"payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:19.840308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:19.840219 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:20.161590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:20.161561 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:24:20.162255 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:24:20.162225 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e5c8e7_d143_4c30_aa75_2a1215c29b1c.slice/crio-86b391a02574492b249aa587a3560524b49dc0662e7368e57a9417323dfee5e9 WatchSource:0}: Error finding container 86b391a02574492b249aa587a3560524b49dc0662e7368e57a9417323dfee5e9: Status 404 returned error can't find the container with id 86b391a02574492b249aa587a3560524b49dc0662e7368e57a9417323dfee5e9 Oct 30 03:24:20.309076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:20.309037 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" event={"ID":"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c","Type":"ContainerStarted","Data":"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16"} Oct 30 03:24:20.309231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:20.309138 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:20.309231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:20.309163 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" event={"ID":"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c","Type":"ContainerStarted","Data":"86b391a02574492b249aa587a3560524b49dc0662e7368e57a9417323dfee5e9"} Oct 30 03:24:20.370084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:20.370023 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" podStartSLOduration=1.370004698 podStartE2EDuration="1.370004698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:20.369221041 +0000 UTC m=+1288.354340938" watchObservedRunningTime="2026-10-30 03:24:20.370004698 +0000 UTC m=+1288.355124586" Oct 30 03:24:23.800577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.800538 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:24:23.806363 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.806340 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.827208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.827172 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:24:23.834693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.834668 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-efbbc5-openshift-default-dockercfg-n5f2l\"" Oct 30 03:24:23.945206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945299 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945370 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945413 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945461 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945559 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945623 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsll\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945692 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:23.945808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:23.945766 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047498 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047577 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsll\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047644 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047787 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047878 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047914 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047952 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.048096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.047999 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.058630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.048940 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.058630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.053928 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.058630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.054815 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.058630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.055050 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.058630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.055294 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.069559 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.065299 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.074537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.070188 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.081537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.077697 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsll\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.101045 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.101009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\") pod \"e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.129832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.129790 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:24.473026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.472990 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:24:24.474506 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:24:24.474480 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87580e59_cc07_4381_a815_e4bd720c8b68.slice/crio-0c37f7bdeddf658f9f6f867b9c39d6a4f7ff12d450a40203cfe4a5430922334a WatchSource:0}: Error finding container 0c37f7bdeddf658f9f6f867b9c39d6a4f7ff12d450a40203cfe4a5430922334a: Status 404 returned error can't find the container with id 0c37f7bdeddf658f9f6f867b9c39d6a4f7ff12d450a40203cfe4a5430922334a Oct 30 03:24:24.477349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.477318 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:24.477449 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.477381 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:24.477449 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:24.477410 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:25.338641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:25.338608 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerStarted","Data":"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd"} Oct 30 03:24:25.339152 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:25.338702 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerStarted","Data":"0c37f7bdeddf658f9f6f867b9c39d6a4f7ff12d450a40203cfe4a5430922334a"} Oct 30 03:24:25.396079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:25.396025 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podStartSLOduration=2.39601233 podStartE2EDuration="2.39601233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:25.393844513 +0000 UTC m=+1293.378964436" watchObservedRunningTime="2026-10-30 03:24:25.39601233 +0000 UTC m=+1293.381132273" Oct 30 03:24:26.130630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:26.130599 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:26.136796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:26.136771 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:26.345114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:26.345084 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:26.345884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:26.345864 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:24:29.691396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.691364 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:24:29.695741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.695716 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.716823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.716794 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:24:29.717053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.717035 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efbbc5\"" Oct 30 03:24:29.726457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.726425 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efbbc5-dockercfg-jtp2k\"" Oct 30 03:24:29.821119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.821078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.821334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.821174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m65g\" (UniqueName: \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.922925 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.922883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.923144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.922969 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m65g\" (UniqueName: \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.923680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.923657 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:29.943598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:29.943529 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m65g\" (UniqueName: \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\") pod \"payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:30.010508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:30.010368 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:30.323700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:30.323670 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:24:30.328762 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:24:30.328727 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a828b55_1179_45cc_98ff_f536e33709f3.slice/crio-2f5731de901ad1bbedfa5d25e2c4520e46304af2b3cccdbb5bc1f6dac784d2d9 WatchSource:0}: Error finding container 2f5731de901ad1bbedfa5d25e2c4520e46304af2b3cccdbb5bc1f6dac784d2d9: Status 404 returned error can't find the container with id 2f5731de901ad1bbedfa5d25e2c4520e46304af2b3cccdbb5bc1f6dac784d2d9 Oct 30 03:24:30.367983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:30.367950 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" event={"ID":"4a828b55-1179-45cc-98ff-f536e33709f3","Type":"ContainerStarted","Data":"2f5731de901ad1bbedfa5d25e2c4520e46304af2b3cccdbb5bc1f6dac784d2d9"} Oct 30 03:24:31.317938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:31.317906 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:24:31.377111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:31.377075 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" event={"ID":"4a828b55-1179-45cc-98ff-f536e33709f3","Type":"ContainerStarted","Data":"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843"} Oct 30 03:24:31.377861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:31.377843 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:31.418975 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:31.418896 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" podStartSLOduration=2.418881251 podStartE2EDuration="2.418881251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:31.416971636 +0000 UTC m=+1299.402091527" watchObservedRunningTime="2026-10-30 03:24:31.418881251 +0000 UTC m=+1299.404001138" Oct 30 03:24:41.755596 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:24:41.754762 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:24:43.396507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:43.396471 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:24:49.537018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.536972 4560 generic.go:356] "Generic (PLEG): container finished" podID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerID="372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062" exitCode=0 Oct 30 03:24:49.537472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.537024 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerDied","Data":"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062"} Oct 30 03:24:49.537472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.537460 4560 scope.go:122] "RemoveContainer" containerID="372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062" Oct 30 03:24:49.539142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.539115 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:49.539234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.539184 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:49.539234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:49.539217 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:24:50.545761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:50.545720 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerStarted","Data":"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb"} Oct 30 03:24:51.415196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:51.415157 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:24:51.416379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:51.416343 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:24:51.416547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:51.416404 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 03:24:52.419540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:52.417966 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:24:52.419540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:52.418056 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 03:24:53.418541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:53.417902 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:24:53.418541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:53.417963 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 03:24:54.418924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:54.418886 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:24:54.419558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:54.419503 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 03:24:55.414696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:55.414663 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:24:55.414956 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:55.414728 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 03:24:56.122091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:56.122038 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:24:56.443444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:56.443348 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:24:56.647766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:24:56.647736 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:25:01.705865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.705829 4560 generic.go:356] "Generic (PLEG): container finished" podID="35751e1f-833d-490b-9464-fd65124c9197" containerID="75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f" exitCode=0 Oct 30 03:25:01.706293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.705912 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerDied","Data":"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f"} Oct 30 03:25:01.706335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.706310 4560 scope.go:122] "RemoveContainer" containerID="75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f" Oct 30 03:25:01.707843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.707819 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:01.707913 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.707881 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:01.707950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:01.707914 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:02.279839 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:02.279797 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:25:02.715981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:02.715939 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerStarted","Data":"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7"} Oct 30 03:25:02.844938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:02.844907 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:25:02.846132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:02.846106 4560 patch_prober.go:29] interesting pod/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 03:25:02.846255 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:02.846152 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 03:25:03.845423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:03.845388 4560 patch_prober.go:29] interesting pod/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 03:25:03.845838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:03.845443 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 03:25:04.845265 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:04.845229 4560 patch_prober.go:29] interesting pod/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 03:25:04.845461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:04.845284 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 03:25:05.845107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:05.845074 4560 patch_prober.go:29] interesting pod/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 03:25:05.845305 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:05.845122 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 03:25:06.845617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:06.845579 4560 patch_prober.go:29] interesting pod/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 03:25:06.845998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:06.845641 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 03:25:07.849335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:07.849304 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:25:07.849850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:07.849719 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:25:07.850276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:07.850256 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:25:11.782740 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:25:11.782698 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:25:17.295792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:17.295756 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:25:17.296256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:17.296045 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" containerID="cri-o://1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa" gracePeriod=30 Oct 30 03:25:19.129720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:19.129685 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:25:19.742534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:19.742483 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:25:19.742723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:19.742567 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:25:20.860965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.860931 4560 generic.go:356] "Generic (PLEG): container finished" podID="87580e59-cc07-4381-a815-e4bd720c8b68" containerID="4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd" exitCode=0 Oct 30 03:25:20.861567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.861009 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerDied","Data":"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd"} Oct 30 03:25:20.861567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.861437 4560 scope.go:122] "RemoveContainer" containerID="4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd" Oct 30 03:25:20.862814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.862780 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:20.862926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.862852 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:20.862926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:20.862884 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:25:21.887469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:21.887433 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerStarted","Data":"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8"} Oct 30 03:25:22.130857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.130821 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:25:22.132223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.132194 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:22.132360 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.132250 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:22.602869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.602848 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:25:22.799784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.799746 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.799978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.799796 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.799978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.799839 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800004 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800073 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800075 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:22.800260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800211 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800219 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:22.800349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800288 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800326 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800388 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:22.800456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800394 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\") pod \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\" (UID: \"c2c7f463-e0d5-4db5-860e-390bdd9d018a\") " Oct 30 03:25:22.800718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800690 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:22.800922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800899 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:22.800962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800937 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.800995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800959 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.800995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800976 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.800995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.800990 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.802216 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.802187 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "kube-api-access-qd8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:22.802328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.802228 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:22.802376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.802329 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:22.803284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.803265 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo" pod "c2c7f463-e0d5-4db5-860e-390bdd9d018a" (UID: "c2c7f463-e0d5-4db5-860e-390bdd9d018a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:25:22.894442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.894358 4560 generic.go:356] "Generic (PLEG): container finished" podID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerID="1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa" exitCode=0 Oct 30 03:25:22.894442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.894413 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" Oct 30 03:25:22.894966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.894443 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerDied","Data":"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa"} Oct 30 03:25:22.894966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.894509 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5" event={"ID":"c2c7f463-e0d5-4db5-860e-390bdd9d018a","Type":"ContainerDied","Data":"ff607cdc6ced7b2b5b203afe79a9fc92aeae27bd282604d2116ba47cf18ea60c"} Oct 30 03:25:22.894966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.894546 4560 scope.go:122] "RemoveContainer" containerID="1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa" Oct 30 03:25:22.901797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.901771 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.901797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.901795 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd8lf\" (UniqueName: \"kubernetes.io/projected/c2c7f463-e0d5-4db5-860e-390bdd9d018a-kube-api-access-qd8lf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.901985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.901808 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.901985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.901822 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.901985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.901835 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c7f463-e0d5-4db5-860e-390bdd9d018a-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:22.904578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.904557 4560 scope.go:122] "RemoveContainer" containerID="af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6" Oct 30 03:25:22.913101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.913084 4560 scope.go:122] "RemoveContainer" containerID="1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa" Oct 30 03:25:22.913351 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:25:22.913328 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa\": container with ID starting with 1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa not found: ID does not exist" containerID="1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa" Oct 30 03:25:22.913436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.913357 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa"} err="failed to get container status \"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa\": rpc error: code = NotFound desc = could not find container \"1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa\": container with ID starting with 1e1cae40f320698539e7a7318dc0b16f8bc2b2323057f062dad4b59205c706aa not found: ID does not exist" Oct 30 03:25:22.913436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.913375 4560 scope.go:122] "RemoveContainer" containerID="af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6" Oct 30 03:25:22.913675 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:25:22.913657 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6\": container with ID starting with af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6 not found: ID does not exist" containerID="af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6" Oct 30 03:25:22.913724 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.913679 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6"} err="failed to get container status \"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6\": rpc error: code = NotFound desc = could not find container \"af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6\": container with ID starting with af1ec6107c58fa11b6a7659b840697cdeb9c8a0e257d4050409101aec3353cd6 not found: ID does not exist" Oct 30 03:25:22.946459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.946433 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:25:22.975099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:22.975067 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0f003-openshift-default-7f5975bd9c-zd7h5"] Oct 30 03:25:23.130227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:23.130194 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:23.130387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:23.130249 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:24.130715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.130661 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" podUID="4a828b55-1179-45cc-98ff-f536e33709f3" containerName="payload-processing" containerID="cri-o://0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843" gracePeriod=25 Oct 30 03:25:24.131148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.130978 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:24.131148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.131038 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:24.176395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.176340 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" path="/var/lib/kubelet/pods/c2c7f463-e0d5-4db5-860e-390bdd9d018a/volumes" Oct 30 03:25:24.440353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.440318 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:25:24.518260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.518220 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\" (UniqueName: \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\") pod \"4a828b55-1179-45cc-98ff-f536e33709f3\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " Oct 30 03:25:24.518450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.518403 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\") pod \"4a828b55-1179-45cc-98ff-f536e33709f3\" (UID: \"4a828b55-1179-45cc-98ff-f536e33709f3\") " Oct 30 03:25:24.519043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.519012 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume" pod "4a828b55-1179-45cc-98ff-f536e33709f3" (UID: "4a828b55-1179-45cc-98ff-f536e33709f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:24.520497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.520464 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g" pod "4a828b55-1179-45cc-98ff-f536e33709f3" (UID: "4a828b55-1179-45cc-98ff-f536e33709f3"). InnerVolumeSpecName "kube-api-access-5m65g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:24.619773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.619728 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a828b55-1179-45cc-98ff-f536e33709f3-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:24.619773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.619758 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m65g\" (UniqueName: \"kubernetes.io/projected/4a828b55-1179-45cc-98ff-f536e33709f3-kube-api-access-5m65g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:25:24.908384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.908347 4560 generic.go:356] "Generic (PLEG): container finished" podID="4a828b55-1179-45cc-98ff-f536e33709f3" containerID="0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843" exitCode=0 Oct 30 03:25:24.908623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.908406 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" Oct 30 03:25:24.908623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.908435 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" event={"ID":"4a828b55-1179-45cc-98ff-f536e33709f3","Type":"ContainerDied","Data":"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843"} Oct 30 03:25:24.908623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.908555 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4" event={"ID":"4a828b55-1179-45cc-98ff-f536e33709f3","Type":"ContainerDied","Data":"2f5731de901ad1bbedfa5d25e2c4520e46304af2b3cccdbb5bc1f6dac784d2d9"} Oct 30 03:25:24.908623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.908590 4560 scope.go:122] "RemoveContainer" containerID="0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843" Oct 30 03:25:24.918377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.918360 4560 scope.go:122] "RemoveContainer" containerID="0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843" Oct 30 03:25:24.918667 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:25:24.918646 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843\": container with ID starting with 0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843 not found: ID does not exist" containerID="0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843" Oct 30 03:25:24.918749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.918679 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843"} err="failed to get container status \"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843\": rpc error: code = NotFound desc = could not find container \"0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843\": container with ID starting with 0d1abf9ba44c4b79eade27ee31e93cb2ba8455b2f64aaa76f0877f946db55843 not found: ID does not exist" Oct 30 03:25:24.977475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.977431 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:25:24.985219 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:24.985186 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-766bc8844-26ln4"] Oct 30 03:25:25.130407 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:25.130368 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:25.130617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:25.130424 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:26.130803 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:26.130772 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:26.131177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:26.130825 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:26.177672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:26.177633 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a828b55-1179-45cc-98ff-f536e33709f3" path="/var/lib/kubelet/pods/4a828b55-1179-45cc-98ff-f536e33709f3/volumes" Oct 30 03:25:26.345364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:26.345328 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:25:27.131070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:27.131037 4560 patch_prober.go:29] interesting pod/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:25:27.131442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:27.131092 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 03:25:28.135404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:28.135375 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:25:28.136431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:28.136408 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:25:33.035353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:33.035306 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7"] Oct 30 03:25:33.047725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:33.047698 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23a9cc7-m26v7"] Oct 30 03:25:34.152991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:34.152962 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e987b2-65e9-4b55-a91c-79c30fb4ded1" path="/var/lib/kubelet/pods/32e987b2-65e9-4b55-a91c-79c30fb4ded1/volumes" Oct 30 03:25:41.749472 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:25:41.749431 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:25:43.470713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.470668 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:25:43.471415 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471389 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:43.471561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471416 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:43.471561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471438 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4a828b55-1179-45cc-98ff-f536e33709f3" containerName="payload-processing" Oct 30 03:25:43.471561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471446 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a828b55-1179-45cc-98ff-f536e33709f3" containerName="payload-processing" Oct 30 03:25:43.471738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471647 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:43.471738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471663 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:43.471738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.471673 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a828b55-1179-45cc-98ff-f536e33709f3" containerName="payload-processing" Oct 30 03:25:43.474963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.474942 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.494180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.494151 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:25:43.506845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.506816 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:25:43.507307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.507285 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4d9a1a-serving-cert\"" Oct 30 03:25:43.507432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.507403 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 03:25:43.508258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.508158 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8gxpp\"" Oct 30 03:25:43.510402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.510381 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:25:43.566297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.566265 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4g5\" (UniqueName: \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.566465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.566336 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.566465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.566438 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.566558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.566495 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.667610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.667572 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.667815 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.667659 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.667815 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.667726 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.667939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.667873 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4g5\" (UniqueName: \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.671095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.671059 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.672826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.672802 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.673019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.672875 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.691911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.691875 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4g5\" (UniqueName: \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\") pod \"maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:43.788438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:43.788351 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:44.088345 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:44.088320 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:25:44.089760 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:25:44.089731 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e97b6b_b314_4070_8f78_e315c76053cd.slice/crio-6ef708262b1a0c3009d2c68538fd3d5a87a4effcf5c064d2c55dc0405af58472 WatchSource:0}: Error finding container 6ef708262b1a0c3009d2c68538fd3d5a87a4effcf5c064d2c55dc0405af58472: Status 404 returned error can't find the container with id 6ef708262b1a0c3009d2c68538fd3d5a87a4effcf5c064d2c55dc0405af58472 Oct 30 03:25:44.091234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:44.091216 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:25:45.028271 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:45.028231 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" event={"ID":"e8e97b6b-b314-4070-8f78-e315c76053cd","Type":"ContainerStarted","Data":"6ef708262b1a0c3009d2c68538fd3d5a87a4effcf5c064d2c55dc0405af58472"} Oct 30 03:25:47.050015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:47.049979 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" event={"ID":"e8e97b6b-b314-4070-8f78-e315c76053cd","Type":"ContainerStarted","Data":"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e"} Oct 30 03:25:47.050505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:47.050082 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:47.096472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:47.096413 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" podStartSLOduration=1.678563802 podStartE2EDuration="4.096398593s" totalImagesPullingTime="2.417834791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:47.09386961 +0000 UTC m=+1375.078989498" watchObservedRunningTime="2026-10-30 03:25:47.096398593 +0000 UTC m=+1375.081518498" Oct 30 03:25:53.061022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:53.060992 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:25:58.334145 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.334107 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:25:58.334977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.334948 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:58.334977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.334972 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c7f463-e0d5-4db5-860e-390bdd9d018a" containerName="istio-proxy" Oct 30 03:25:58.340580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.340556 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.362047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.362009 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:25:58.369816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.369781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6c106c\"/\"openshift-service-ca.crt\"" Oct 30 03:25:58.370003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.369828 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6c106c\"/\"kube-root-ca.crt\"" Oct 30 03:25:58.370003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.369935 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6c106c\"/\"default-dockercfg-7kt9x\"" Oct 30 03:25:58.370297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.370272 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6c106c\"/\"ipp-route-6c106c-kserve-self-signed-certs\"" Oct 30 03:25:58.446001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.445958 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.446159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.446086 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.446235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.446211 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.446272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.446255 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.446379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.446358 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.446433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.446398 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slthz\" (UniqueName: \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.547879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.547841 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.547917 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548024 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548063 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548115 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548143 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slthz\" (UniqueName: \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548306 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548410 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.548592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.548574 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.550258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.550231 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.550921 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.550897 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.573884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.573241 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slthz\" (UniqueName: \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\") pod \"ipp-route-6c106c-kserve-97b7559d5-9wwvp\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.657283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.657178 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:25:58.960736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:58.960710 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:25:58.962657 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:25:58.962624 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b5a9be5_136e_406b_9ff9_4e7aad25503c.slice/crio-8de7416f041fba7ff1398a2d86023696847699a3480160dd782ddd91c0eb2c3a WatchSource:0}: Error finding container 8de7416f041fba7ff1398a2d86023696847699a3480160dd782ddd91c0eb2c3a: Status 404 returned error can't find the container with id 8de7416f041fba7ff1398a2d86023696847699a3480160dd782ddd91c0eb2c3a Oct 30 03:25:59.120290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:59.120246 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" event={"ID":"6b5a9be5-136e-406b-9ff9-4e7aad25503c","Type":"ContainerStarted","Data":"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3"} Oct 30 03:25:59.120290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:59.120295 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" event={"ID":"6b5a9be5-136e-406b-9ff9-4e7aad25503c","Type":"ContainerStarted","Data":"8de7416f041fba7ff1398a2d86023696847699a3480160dd782ddd91c0eb2c3a"} Oct 30 03:25:59.161267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:59.161217 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" podStartSLOduration=1.161202764 podStartE2EDuration="1.161202764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:59.159216754 +0000 UTC m=+1387.144336643" watchObservedRunningTime="2026-10-30 03:25:59.161202764 +0000 UTC m=+1387.146322653" Oct 30 03:25:59.657804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:25:59.657773 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:26:08.658266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:08.658171 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:26:08.671549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:08.671503 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:26:09.193883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:09.193850 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:26:11.749603 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:26:11.749564 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:26:12.963968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:12.963931 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:26:12.968155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:12.968129 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:12.985745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:12.985715 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:26:12.995269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:12.995238 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3eea00\"" Oct 30 03:26:13.005608 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.005581 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3eea00-dockercfg-wkklb\"" Oct 30 03:26:13.082077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.081749 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5dr\" (UniqueName: \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.082077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.081876 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.109575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.104126 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:26:13.110242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.110215 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:26:13.110550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.110464 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.182645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.182612 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5dr\" (UniqueName: \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.182843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.182653 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs78h\" (UniqueName: \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.182843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.182742 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.182961 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.182884 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.183448 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.183415 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.202666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.202631 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5dr\" (UniqueName: \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\") pod \"payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.283294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.283208 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:13.283604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.283576 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs78h\" (UniqueName: \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.283786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.283768 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.284487 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.284467 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.303892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.303760 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs78h\" (UniqueName: \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\") pod \"payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.466941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.466910 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:13.589013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.588986 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:26:13.590184 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:26:13.590160 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf53baa_dfaa_4241_9278_4449eb2f25ad.slice/crio-e2c29074e812091cb81ad6b118ebb96906d8562ad0f0a88c79cfcf11bea9c988 WatchSource:0}: Error finding container e2c29074e812091cb81ad6b118ebb96906d8562ad0f0a88c79cfcf11bea9c988: Status 404 returned error can't find the container with id e2c29074e812091cb81ad6b118ebb96906d8562ad0f0a88c79cfcf11bea9c988 Oct 30 03:26:13.769963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:13.769933 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:26:13.771118 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:26:13.771094 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d77d7a7_48d0_4a10_afff_8697d30f8276.slice/crio-c58a2f52d4db1938586a307ba259782c64d03725cc7b30910f2cbe40e7816fde WatchSource:0}: Error finding container c58a2f52d4db1938586a307ba259782c64d03725cc7b30910f2cbe40e7816fde: Status 404 returned error can't find the container with id c58a2f52d4db1938586a307ba259782c64d03725cc7b30910f2cbe40e7816fde Oct 30 03:26:14.248777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.248487 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" event={"ID":"1d77d7a7-48d0-4a10-afff-8697d30f8276","Type":"ContainerStarted","Data":"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b"} Oct 30 03:26:14.248777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.248567 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" event={"ID":"1d77d7a7-48d0-4a10-afff-8697d30f8276","Type":"ContainerStarted","Data":"c58a2f52d4db1938586a307ba259782c64d03725cc7b30910f2cbe40e7816fde"} Oct 30 03:26:14.248777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.248598 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:14.250379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.250350 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" event={"ID":"bbf53baa-dfaa-4241-9278-4449eb2f25ad","Type":"ContainerStarted","Data":"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667"} Oct 30 03:26:14.250528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.250398 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" event={"ID":"bbf53baa-dfaa-4241-9278-4449eb2f25ad","Type":"ContainerStarted","Data":"e2c29074e812091cb81ad6b118ebb96906d8562ad0f0a88c79cfcf11bea9c988"} Oct 30 03:26:14.250528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.250431 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:14.289629 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:14.289565 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" podStartSLOduration=1.289546069 podStartE2EDuration="1.289546069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:14.287629518 +0000 UTC m=+1402.272749407" watchObservedRunningTime="2026-10-30 03:26:14.289546069 +0000 UTC m=+1402.274665958" Oct 30 03:26:23.758467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:23.758413 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" podStartSLOduration=11.75839737 podStartE2EDuration="11.75839737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:14.332098668 +0000 UTC m=+1402.317218559" watchObservedRunningTime="2026-10-30 03:26:23.75839737 +0000 UTC m=+1411.743517261" Oct 30 03:26:23.761191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:23.761161 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:26:25.270663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:25.270632 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:26:25.271117 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:25.270697 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:26:28.762185 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:28.762138 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" podUID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" containerName="payload-pre-processing" containerID="cri-o://0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16" gracePeriod=25 Oct 30 03:26:29.028149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.028085 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:26:29.113041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.113008 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\" (UniqueName: \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\") pod \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " Oct 30 03:26:29.113185 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.113135 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\") pod \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\" (UID: \"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c\") " Oct 30 03:26:29.113605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.113574 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume" pod "a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" (UID: "a2e5c8e7-d143-4c30-aa75-2a1215c29b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:29.115366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.115334 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl" pod "a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" (UID: "a2e5c8e7-d143-4c30-aa75-2a1215c29b1c"). InnerVolumeSpecName "kube-api-access-dr6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:29.214814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.214782 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6dl\" (UniqueName: \"kubernetes.io/projected/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-kube-api-access-dr6dl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:29.214814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.214811 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:29.365467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.365385 4560 generic.go:356] "Generic (PLEG): container finished" podID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" containerID="0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16" exitCode=0 Oct 30 03:26:29.365467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.365428 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" Oct 30 03:26:29.365467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.365450 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" event={"ID":"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c","Type":"ContainerDied","Data":"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16"} Oct 30 03:26:29.365753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.365494 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn" event={"ID":"a2e5c8e7-d143-4c30-aa75-2a1215c29b1c","Type":"ContainerDied","Data":"86b391a02574492b249aa587a3560524b49dc0662e7368e57a9417323dfee5e9"} Oct 30 03:26:29.365753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.365550 4560 scope.go:122] "RemoveContainer" containerID="0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16" Oct 30 03:26:29.375084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.375060 4560 scope.go:122] "RemoveContainer" containerID="0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16" Oct 30 03:26:29.375362 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:26:29.375341 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16\": container with ID starting with 0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16 not found: ID does not exist" containerID="0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16" Oct 30 03:26:29.375430 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.375370 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16"} err="failed to get container status \"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16\": rpc error: code = NotFound desc = could not find container \"0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16\": container with ID starting with 0e325d37553cb46eb39d17686cfd332c92df0fcd24b6f242336917064caf9b16 not found: ID does not exist" Oct 30 03:26:29.418685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.418625 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:26:29.430797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:29.430765 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dff241-747567f57f-gjtkn"] Oct 30 03:26:30.152438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:30.152404 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" path="/var/lib/kubelet/pods/a2e5c8e7-d143-4c30-aa75-2a1215c29b1c/volumes" Oct 30 03:26:34.651491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:34.651450 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:26:34.651987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:34.651732 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" containerID="cri-o://04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb" gracePeriod=30 Oct 30 03:26:39.921316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.921256 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:26:39.973303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973265 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973336 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973406 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973476 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973509 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973567 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973685 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973745 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.973865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:39.973865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973802 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:39.973960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.973880 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974064 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\") pod \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\" (UID: \"2a9c94d9-ebcf-4d7d-a1e4-399435057536\") " Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974157 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974243 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974486 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974500 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974509 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974546 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:39.974583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.974558 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:39.975839 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.975814 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:39.976120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.976095 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:39.976236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.976117 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "kube-api-access-ffmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:39.976236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:39.976198 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo" pod "2a9c94d9-ebcf-4d7d-a1e4-399435057536" (UID: "2a9c94d9-ebcf-4d7d-a1e4-399435057536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:26:40.075931 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.075892 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:40.075931 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.075928 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:40.075931 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.075939 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:40.076197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.075951 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmrk\" (UniqueName: \"kubernetes.io/projected/2a9c94d9-ebcf-4d7d-a1e4-399435057536-kube-api-access-ffmrk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:26:40.430572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.430539 4560 generic.go:356] "Generic (PLEG): container finished" podID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerID="04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb" exitCode=0 Oct 30 03:26:40.430756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.430638 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" Oct 30 03:26:40.430756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.430681 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerDied","Data":"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb"} Oct 30 03:26:40.430756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.430737 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b" event={"ID":"2a9c94d9-ebcf-4d7d-a1e4-399435057536","Type":"ContainerDied","Data":"d5a93fd41acf285f5f8ca492ce1a72ac0faca5950093f2bdbbd169e3847573eb"} Oct 30 03:26:40.430854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.430760 4560 scope.go:122] "RemoveContainer" containerID="04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb" Oct 30 03:26:40.440244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.440224 4560 scope.go:122] "RemoveContainer" containerID="372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062" Oct 30 03:26:40.449038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.449019 4560 scope.go:122] "RemoveContainer" containerID="04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb" Oct 30 03:26:40.449319 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:26:40.449300 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb\": container with ID starting with 04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb not found: ID does not exist" containerID="04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb" Oct 30 03:26:40.449379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.449328 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb"} err="failed to get container status \"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb\": rpc error: code = NotFound desc = could not find container \"04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb\": container with ID starting with 04a25a86c991c86cc7ed7b88f0986aea639b7c3531962bafe16d42b1cebd8adb not found: ID does not exist" Oct 30 03:26:40.449379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.449345 4560 scope.go:122] "RemoveContainer" containerID="372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062" Oct 30 03:26:40.449717 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:26:40.449697 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062\": container with ID starting with 372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062 not found: ID does not exist" containerID="372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062" Oct 30 03:26:40.449778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.449725 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062"} err="failed to get container status \"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062\": rpc error: code = NotFound desc = could not find container \"372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062\": container with ID starting with 372be9e10bb4bce75d4b263f931ff64d527fc5480d75b8b6c86becd22c131062 not found: ID does not exist" Oct 30 03:26:40.494044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.494004 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:26:40.507381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:40.507351 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b03a5e-openshift-default-6b4c88bd4f-mb29b"] Oct 30 03:26:41.756235 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:26:41.756192 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:26:41.967139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967107 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:26:41.967644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967630 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" containerName="payload-pre-processing" Oct 30 03:26:41.967694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967645 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" containerName="payload-pre-processing" Oct 30 03:26:41.967694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967667 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:41.967694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967672 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:41.967694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967680 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:41.967694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967685 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:41.967847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967776 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:41.967847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.967788 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e5c8e7-d143-4c30-aa75-2a1215c29b1c" containerName="payload-pre-processing" Oct 30 03:26:41.972705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.972685 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:41.989215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.989185 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:26:41.999112 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:41.999082 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-194506\"" Oct 30 03:26:42.044317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.044219 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-194506-dockercfg-qs67f\"" Oct 30 03:26:42.108025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.104346 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:26:42.108025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.105290 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" containerName="istio-proxy" Oct 30 03:26:42.110061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.109191 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:26:42.110061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.109455 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.114706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.114683 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srv9p\" (UniqueName: \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.131917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.131884 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.155790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.155758 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9c94d9-ebcf-4d7d-a1e4-399435057536" path="/var/lib/kubelet/pods/2a9c94d9-ebcf-4d7d-a1e4-399435057536/volumes" Oct 30 03:26:42.233159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.233122 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftflb\" (UniqueName: \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.233341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.233289 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.233444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.233428 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srv9p\" (UniqueName: \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.233489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.233456 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.234095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.234073 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.252957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.252924 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srv9p\" (UniqueName: \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\") pod \"payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.286896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.286868 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:42.334636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.334593 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.335548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.335478 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.335626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.335608 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftflb\" (UniqueName: \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.361819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.361783 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftflb\" (UniqueName: \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\") pod \"payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.436737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.436698 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:42.596928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.596902 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:26:42.598226 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:26:42.598193 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99daeb88_5777_4162_a4c2_509a984e742e.slice/crio-e05a5fdf1c4faa02f81417a6f7834fa25cfb972accc9c0d0e82ffda29b4a8465 WatchSource:0}: Error finding container e05a5fdf1c4faa02f81417a6f7834fa25cfb972accc9c0d0e82ffda29b4a8465: Status 404 returned error can't find the container with id e05a5fdf1c4faa02f81417a6f7834fa25cfb972accc9c0d0e82ffda29b4a8465 Oct 30 03:26:42.748633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:42.748609 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:26:42.749478 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:26:42.749455 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b91c87_dd7c_4a34_8e5c_e5fabb77afc6.slice/crio-8cdc6648ad69eeb90d575a391750a5b2e8f5ddce8b97611149edde7f557efe88 WatchSource:0}: Error finding container 8cdc6648ad69eeb90d575a391750a5b2e8f5ddce8b97611149edde7f557efe88: Status 404 returned error can't find the container with id 8cdc6648ad69eeb90d575a391750a5b2e8f5ddce8b97611149edde7f557efe88 Oct 30 03:26:43.450150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.450115 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" event={"ID":"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6","Type":"ContainerStarted","Data":"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e"} Oct 30 03:26:43.450608 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.450167 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" event={"ID":"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6","Type":"ContainerStarted","Data":"8cdc6648ad69eeb90d575a391750a5b2e8f5ddce8b97611149edde7f557efe88"} Oct 30 03:26:43.450608 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.450208 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:43.451885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.451862 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" event={"ID":"99daeb88-5777-4162-a4c2-509a984e742e","Type":"ContainerStarted","Data":"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83"} Oct 30 03:26:43.451992 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.451907 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:43.451992 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.451923 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" event={"ID":"99daeb88-5777-4162-a4c2-509a984e742e","Type":"ContainerStarted","Data":"e05a5fdf1c4faa02f81417a6f7834fa25cfb972accc9c0d0e82ffda29b4a8465"} Oct 30 03:26:43.616223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.616163 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" podStartSLOduration=1.6161416800000001 podStartE2EDuration="1.61614168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:43.562730478 +0000 UTC m=+1431.547850366" watchObservedRunningTime="2026-10-30 03:26:43.61614168 +0000 UTC m=+1431.601261569" Oct 30 03:26:43.618544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:43.618474 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" podStartSLOduration=2.618454505 podStartE2EDuration="2.618454505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:43.614625866 +0000 UTC m=+1431.599745779" watchObservedRunningTime="2026-10-30 03:26:43.618454505 +0000 UTC m=+1431.603574430" Oct 30 03:26:52.538279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:52.538249 4560 scope.go:122] "RemoveContainer" containerID="30f9beadedb28a2d69e597a5d56b92163c7f3d2484931b76dde6f0a73dda6c1b" Oct 30 03:26:52.547469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:52.547446 4560 scope.go:122] "RemoveContainer" containerID="a98b6c6d892a49ae5a36eee475314803af5d9f81dbb6042b5e21652dfd373897" Oct 30 03:26:54.465764 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:54.465727 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:26:54.469549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:54.468738 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:26:56.544011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:56.543975 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:26:56.544579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:26:56.544206 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" containerID="cri-o://d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7" gracePeriod=30 Oct 30 03:27:01.812077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.812015 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:27:01.865901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.865858 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.865916 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.865962 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866047 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866107 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866186 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:01.866355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866284 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.866355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866337 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866370 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866375 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866401 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\") pod \"35751e1f-833d-490b-9464-fd65124c9197\" (UID: \"35751e1f-833d-490b-9464-fd65124c9197\") " Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866403 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866712 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866841 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866951 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866969 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866983 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.867174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.866995 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35751e1f-833d-490b-9464-fd65124c9197-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.868639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.867775 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.868639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.868423 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:27:01.869286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.869259 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:01.869456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.869427 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "kube-api-access-xzsp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:01.870754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.870733 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token" pod "35751e1f-833d-490b-9464-fd65124c9197" (UID: "35751e1f-833d-490b-9464-fd65124c9197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:01.971230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.971191 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35751e1f-833d-490b-9464-fd65124c9197-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.971230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.971232 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsp6\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-kube-api-access-xzsp6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.971465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.971243 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35751e1f-833d-490b-9464-fd65124c9197-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:01.971465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:01.971256 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35751e1f-833d-490b-9464-fd65124c9197-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:02.572604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.572565 4560 generic.go:356] "Generic (PLEG): container finished" podID="35751e1f-833d-490b-9464-fd65124c9197" containerID="d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7" exitCode=0 Oct 30 03:27:02.572786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.572623 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" Oct 30 03:27:02.572786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.572652 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerDied","Data":"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7"} Oct 30 03:27:02.572786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.572716 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz" event={"ID":"35751e1f-833d-490b-9464-fd65124c9197","Type":"ContainerDied","Data":"f3a71a23e5087c62d9bbecd37c6a6b558f9e60e2e4e64c7280e25f7e8416b88c"} Oct 30 03:27:02.572786 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.572741 4560 scope.go:122] "RemoveContainer" containerID="d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7" Oct 30 03:27:02.582857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.582838 4560 scope.go:122] "RemoveContainer" containerID="75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f" Oct 30 03:27:02.591544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.591526 4560 scope.go:122] "RemoveContainer" containerID="d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7" Oct 30 03:27:02.591830 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:02.591804 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7\": container with ID starting with d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7 not found: ID does not exist" containerID="d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7" Oct 30 03:27:02.591922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.591835 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7"} err="failed to get container status \"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7\": rpc error: code = NotFound desc = could not find container \"d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7\": container with ID starting with d2e4af2968c5b24e7b9d59347fa12f22eb58cba7f5293c44ceba2731007515e7 not found: ID does not exist" Oct 30 03:27:02.591922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.591853 4560 scope.go:122] "RemoveContainer" containerID="75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f" Oct 30 03:27:02.592083 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:02.592066 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f\": container with ID starting with 75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f not found: ID does not exist" containerID="75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f" Oct 30 03:27:02.592123 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.592090 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f"} err="failed to get container status \"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f\": rpc error: code = NotFound desc = could not find container \"75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f\": container with ID starting with 75099ea2c362978cfce64082eb8d3db98da507b33c34c82715686ed7b9aa4d0f not found: ID does not exist" Oct 30 03:27:02.637898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.637858 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:27:02.641280 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:02.641250 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-dff241-openshift-default-789f678dc7-tgghz"] Oct 30 03:27:04.152618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.152580 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35751e1f-833d-490b-9464-fd65124c9197" path="/var/lib/kubelet/pods/35751e1f-833d-490b-9464-fd65124c9197/volumes" Oct 30 03:27:04.614666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.614633 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:27:04.615197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615183 4560 cpu_manager.go:414] "containerMap: removing container" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.615238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615198 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.615238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615208 4560 cpu_manager.go:414] "containerMap: removing container" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.615238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615213 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.615347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615301 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.615347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.615314 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="35751e1f-833d-490b-9464-fd65124c9197" containerName="istio-proxy" Oct 30 03:27:04.620075 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.620059 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.648849 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.648120 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-194506-serving-cert\"" Oct 30 03:27:04.652772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.652740 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:27:04.704892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.704858 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.705066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.704923 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.705123 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.705051 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw9z\" (UniqueName: \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.705243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.705225 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.807037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.807003 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.807248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.807076 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.807248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.807134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.807248 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.807228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw9z\" (UniqueName: \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.807413 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:04.807280 4560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-serving-cert: secret "maas-api-e2e-praxis-a-194506-serving-cert" not found Oct 30 03:27:04.807413 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:04.807369 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls podName:6e43c67c-693c-4c85-84ed-0f38542ea858 nodeName:}" failed. No retries permitted until 2026-10-30 03:27:05.307348656 +0000 UTC m=+1453.292468541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls") pod "maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" (UID: "6e43c67c-693c-4c85-84ed-0f38542ea858") : secret "maas-api-e2e-praxis-a-194506-serving-cert" not found Oct 30 03:27:04.807813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.807794 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.809620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.809599 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:04.830092 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:04.830056 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw9z\" (UniqueName: \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:05.314364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:05.314334 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:05.316801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:05.316772 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") pod \"maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:05.533539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:05.533489 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:05.852160 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:05.852125 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:27:06.604865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:06.604772 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" event={"ID":"6e43c67c-693c-4c85-84ed-0f38542ea858","Type":"ContainerStarted","Data":"ea6ea71a3a975d5bc8ab23ebae4ed87a21e6e7f6bfca0ae2876773a3b8f78d73"} Oct 30 03:27:06.604865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:06.604831 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" event={"ID":"6e43c67c-693c-4c85-84ed-0f38542ea858","Type":"ContainerStarted","Data":"05da31b3a65315c955178a31da77416fb4a1b31721198234a193f23258eb1bd0"} Oct 30 03:27:06.605330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:06.604873 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:06.645752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:06.645695 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" podStartSLOduration=2.236701997 podStartE2EDuration="2.645676659s" totalImagesPullingTime="408.974662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:06.644654804 +0000 UTC m=+1454.629774714" watchObservedRunningTime="2026-10-30 03:27:06.645676659 +0000 UTC m=+1454.630796548" Oct 30 03:27:11.749178 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:11.749139 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:27:12.615634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:12.615607 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:27:21.714577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:21.714538 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:27:21.715065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:21.714860 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" podUID="e8e97b6b-b314-4070-8f78-e315c76053cd" containerName="maas-api" containerID="cri-o://280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e" gracePeriod=30 Oct 30 03:27:22.030274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.030242 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:27:22.122317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122285 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\" (UniqueName: \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\") pod \"e8e97b6b-b314-4070-8f78-e315c76053cd\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " Oct 30 03:27:22.122509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122429 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\") pod \"e8e97b6b-b314-4070-8f78-e315c76053cd\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " Oct 30 03:27:22.122509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122467 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\") pod \"e8e97b6b-b314-4070-8f78-e315c76053cd\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " Oct 30 03:27:22.122509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122491 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\") pod \"e8e97b6b-b314-4070-8f78-e315c76053cd\" (UID: \"e8e97b6b-b314-4070-8f78-e315c76053cd\") " Oct 30 03:27:22.122862 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122834 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca" pod "e8e97b6b-b314-4070-8f78-e315c76053cd" (UID: "e8e97b6b-b314-4070-8f78-e315c76053cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:22.122984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.122968 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e97b6b-b314-4070-8f78-e315c76053cd-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:22.124562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.124494 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs" pod "e8e97b6b-b314-4070-8f78-e315c76053cd" (UID: "e8e97b6b-b314-4070-8f78-e315c76053cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:22.124670 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.124604 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls" pod "e8e97b6b-b314-4070-8f78-e315c76053cd" (UID: "e8e97b6b-b314-4070-8f78-e315c76053cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:22.124911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.124892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5" pod "e8e97b6b-b314-4070-8f78-e315c76053cd" (UID: "e8e97b6b-b314-4070-8f78-e315c76053cd"). InnerVolumeSpecName "kube-api-access-jr4g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:22.232538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.231644 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4g5\" (UniqueName: \"kubernetes.io/projected/e8e97b6b-b314-4070-8f78-e315c76053cd-kube-api-access-jr4g5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:22.232538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.231684 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:22.232538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.231698 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e97b6b-b314-4070-8f78-e315c76053cd-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:22.705056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.705024 4560 generic.go:356] "Generic (PLEG): container finished" podID="e8e97b6b-b314-4070-8f78-e315c76053cd" containerID="280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e" exitCode=0 Oct 30 03:27:22.705235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.705073 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" Oct 30 03:27:22.705235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.705085 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" event={"ID":"e8e97b6b-b314-4070-8f78-e315c76053cd","Type":"ContainerDied","Data":"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e"} Oct 30 03:27:22.705235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.705128 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f" event={"ID":"e8e97b6b-b314-4070-8f78-e315c76053cd","Type":"ContainerDied","Data":"6ef708262b1a0c3009d2c68538fd3d5a87a4effcf5c064d2c55dc0405af58472"} Oct 30 03:27:22.705235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.705150 4560 scope.go:122] "RemoveContainer" containerID="280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e" Oct 30 03:27:22.714570 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.714546 4560 scope.go:122] "RemoveContainer" containerID="280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e" Oct 30 03:27:22.714855 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:22.714834 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e\": container with ID starting with 280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e not found: ID does not exist" containerID="280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e" Oct 30 03:27:22.715131 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.714865 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e"} err="failed to get container status \"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e\": rpc error: code = NotFound desc = could not find container \"280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e\": container with ID starting with 280fa7d48dd35aa37a885cce34d550d859ad30ce54f9bfd9c3e539edfed30a6e not found: ID does not exist" Oct 30 03:27:22.757609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.757573 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:27:22.770158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:22.770127 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4d9a1a-5d7f5c5b66-ccl9f"] Oct 30 03:27:24.158739 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:24.154294 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e97b6b-b314-4070-8f78-e315c76053cd" path="/var/lib/kubelet/pods/e8e97b6b-b314-4070-8f78-e315c76053cd/volumes" Oct 30 03:27:27.283208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:27.283173 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:27:27.317928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:27.317887 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:27:32.283752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.283706 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" podUID="1d77d7a7-48d0-4a10-afff-8697d30f8276" containerName="payload-processing" containerID="cri-o://78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b" gracePeriod=25 Oct 30 03:27:32.319088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.319027 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" podUID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" containerName="payload-pre-processing" containerID="cri-o://e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667" gracePeriod=25 Oct 30 03:27:32.592087 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.592060 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:27:32.614190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.614160 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:27:32.682860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.682823 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\" (UniqueName: \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\") pod \"1d77d7a7-48d0-4a10-afff-8697d30f8276\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " Oct 30 03:27:32.683032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.682984 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\") pod \"1d77d7a7-48d0-4a10-afff-8697d30f8276\" (UID: \"1d77d7a7-48d0-4a10-afff-8697d30f8276\") " Oct 30 03:27:32.683157 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683138 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\" (UniqueName: \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\") pod \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " Oct 30 03:27:32.683244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683228 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\") pod \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\" (UID: \"bbf53baa-dfaa-4241-9278-4449eb2f25ad\") " Oct 30 03:27:32.683410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683386 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume" pod "1d77d7a7-48d0-4a10-afff-8697d30f8276" (UID: "1d77d7a7-48d0-4a10-afff-8697d30f8276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:32.683666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683646 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume" pod "bbf53baa-dfaa-4241-9278-4449eb2f25ad" (UID: "bbf53baa-dfaa-4241-9278-4449eb2f25ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:32.683893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683872 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf53baa-dfaa-4241-9278-4449eb2f25ad-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:32.683983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.683901 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d77d7a7-48d0-4a10-afff-8697d30f8276-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:32.685561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.685535 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h" pod "1d77d7a7-48d0-4a10-afff-8697d30f8276" (UID: "1d77d7a7-48d0-4a10-afff-8697d30f8276"). InnerVolumeSpecName "kube-api-access-qs78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:32.685682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.685576 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr" pod "bbf53baa-dfaa-4241-9278-4449eb2f25ad" (UID: "bbf53baa-dfaa-4241-9278-4449eb2f25ad"). InnerVolumeSpecName "kube-api-access-vz5dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:32.764031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.763994 4560 generic.go:356] "Generic (PLEG): container finished" podID="1d77d7a7-48d0-4a10-afff-8697d30f8276" containerID="78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b" exitCode=0 Oct 30 03:27:32.764221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.764047 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" Oct 30 03:27:32.764221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.764104 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" event={"ID":"1d77d7a7-48d0-4a10-afff-8697d30f8276","Type":"ContainerDied","Data":"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b"} Oct 30 03:27:32.764221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.764182 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb" event={"ID":"1d77d7a7-48d0-4a10-afff-8697d30f8276","Type":"ContainerDied","Data":"c58a2f52d4db1938586a307ba259782c64d03725cc7b30910f2cbe40e7816fde"} Oct 30 03:27:32.764221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.764216 4560 scope.go:122] "RemoveContainer" containerID="78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b" Oct 30 03:27:32.766595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.766564 4560 generic.go:356] "Generic (PLEG): container finished" podID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" containerID="e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667" exitCode=0 Oct 30 03:27:32.766729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.766631 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" Oct 30 03:27:32.766729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.766654 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" event={"ID":"bbf53baa-dfaa-4241-9278-4449eb2f25ad","Type":"ContainerDied","Data":"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667"} Oct 30 03:27:32.766729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.766692 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj" event={"ID":"bbf53baa-dfaa-4241-9278-4449eb2f25ad","Type":"ContainerDied","Data":"e2c29074e812091cb81ad6b118ebb96906d8562ad0f0a88c79cfcf11bea9c988"} Oct 30 03:27:32.776755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.776737 4560 scope.go:122] "RemoveContainer" containerID="78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b" Oct 30 03:27:32.777168 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:32.777098 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b\": container with ID starting with 78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b not found: ID does not exist" containerID="78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b" Oct 30 03:27:32.777239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.777172 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b"} err="failed to get container status \"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b\": rpc error: code = NotFound desc = could not find container \"78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b\": container with ID starting with 78edba8f44248f95f1655947ca6bde25f020c4984eae8c790ae7b23f6c5a655b not found: ID does not exist" Oct 30 03:27:32.777239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.777198 4560 scope.go:122] "RemoveContainer" containerID="e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667" Oct 30 03:27:32.788834 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.788805 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5dr\" (UniqueName: \"kubernetes.io/projected/bbf53baa-dfaa-4241-9278-4449eb2f25ad-kube-api-access-vz5dr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:32.788976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.788838 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs78h\" (UniqueName: \"kubernetes.io/projected/1d77d7a7-48d0-4a10-afff-8697d30f8276-kube-api-access-qs78h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:32.790422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.790391 4560 scope.go:122] "RemoveContainer" containerID="e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667" Oct 30 03:27:32.790774 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:32.790743 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667\": container with ID starting with e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667 not found: ID does not exist" containerID="e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667" Oct 30 03:27:32.790861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.790787 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667"} err="failed to get container status \"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667\": rpc error: code = NotFound desc = could not find container \"e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667\": container with ID starting with e79f361f8fe53b02a370dbbdbce62337e2ed8cb22cea3a8e81de67e50085c667 not found: ID does not exist" Oct 30 03:27:32.825810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.825679 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:27:32.829108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.829080 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3eea00-6cf79858d7-jpfcj"] Oct 30 03:27:32.868856 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.868820 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:27:32.874953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:32.874925 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3eea00-5b99fb66db-m4fxb"] Oct 30 03:27:33.411385 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:33.411348 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:27:34.154147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:34.154069 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d77d7a7-48d0-4a10-afff-8697d30f8276" path="/var/lib/kubelet/pods/1d77d7a7-48d0-4a10-afff-8697d30f8276/volumes" Oct 30 03:27:34.154542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:34.154512 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" path="/var/lib/kubelet/pods/bbf53baa-dfaa-4241-9278-4449eb2f25ad/volumes" Oct 30 03:27:41.750423 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:41.750384 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:27:48.432963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.432920 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" podUID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" containerName="main" containerID="cri-o://64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3" gracePeriod=45 Oct 30 03:27:48.686935 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.686875 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:27:48.772500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772461 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.772697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772525 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.772697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772585 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\" (UniqueName: \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.772697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772675 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.772871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772707 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.772871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772779 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:48.772871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\") pod \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\" (UID: \"6b5a9be5-136e-406b-9ff9-4e7aad25503c\") " Oct 30 03:27:48.773030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.772876 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:48.773125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.773105 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:48.773256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.773238 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.773385 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.773257 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.773385 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.773302 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.775288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.775260 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "kube-api-access-slthz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:48.775410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.775348 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:48.775410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.775397 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs" pod "6b5a9be5-136e-406b-9ff9-4e7aad25503c" (UID: "6b5a9be5-136e-406b-9ff9-4e7aad25503c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:48.866459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.866425 4560 generic.go:356] "Generic (PLEG): container finished" podID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" containerID="64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3" exitCode=0 Oct 30 03:27:48.866644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.866470 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" Oct 30 03:27:48.866644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.866500 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" event={"ID":"6b5a9be5-136e-406b-9ff9-4e7aad25503c","Type":"ContainerDied","Data":"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3"} Oct 30 03:27:48.866644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.866553 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp" event={"ID":"6b5a9be5-136e-406b-9ff9-4e7aad25503c","Type":"ContainerDied","Data":"8de7416f041fba7ff1398a2d86023696847699a3480160dd782ddd91c0eb2c3a"} Oct 30 03:27:48.866644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.866571 4560 scope.go:122] "RemoveContainer" containerID="64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3" Oct 30 03:27:48.875088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.875067 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5a9be5-136e-406b-9ff9-4e7aad25503c-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.875197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.875095 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slthz\" (UniqueName: \"kubernetes.io/projected/6b5a9be5-136e-406b-9ff9-4e7aad25503c-kube-api-access-slthz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.875197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.875109 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5a9be5-136e-406b-9ff9-4e7aad25503c-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:27:48.876696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.876681 4560 scope.go:122] "RemoveContainer" containerID="64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3" Oct 30 03:27:48.876975 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:27:48.876950 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3\": container with ID starting with 64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3 not found: ID does not exist" containerID="64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3" Oct 30 03:27:48.877046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.876983 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3"} err="failed to get container status \"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3\": rpc error: code = NotFound desc = could not find container \"64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3\": container with ID starting with 64d5be09dac03f8dc5d92111faff6b44c1293026699fcefe17fb20486759f5d3 not found: ID does not exist" Oct 30 03:27:48.919144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.919110 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:27:48.921263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:48.921242 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6c106c/ipp-route-6c106c-kserve-97b7559d5-9wwvp"] Oct 30 03:27:50.152845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:50.152816 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" path="/var/lib/kubelet/pods/6b5a9be5-136e-406b-9ff9-4e7aad25503c/volumes" Oct 30 03:27:59.257493 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:27:59.257454 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" podUID="b6ab5ae10a035469fb072566acf5cdc5" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 03:28:11.752163 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:28:11.752126 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:28:11.845772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.845740 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z"] Oct 30 03:28:11.846296 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846282 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e8e97b6b-b314-4070-8f78-e315c76053cd" containerName="maas-api" Oct 30 03:28:11.846339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846297 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e97b6b-b314-4070-8f78-e315c76053cd" containerName="maas-api" Oct 30 03:28:11.846339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846306 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" containerName="payload-pre-processing" Oct 30 03:28:11.846339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846311 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" containerName="payload-pre-processing" Oct 30 03:28:11.846339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846328 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1d77d7a7-48d0-4a10-afff-8697d30f8276" containerName="payload-processing" Oct 30 03:28:11.846339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846332 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d77d7a7-48d0-4a10-afff-8697d30f8276" containerName="payload-processing" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846346 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" containerName="main" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846351 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" containerName="main" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846421 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d77d7a7-48d0-4a10-afff-8697d30f8276" containerName="payload-processing" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846431 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5a9be5-136e-406b-9ff9-4e7aad25503c" containerName="main" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846438 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e97b6b-b314-4070-8f78-e315c76053cd" containerName="maas-api" Oct 30 03:28:11.846490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.846445 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf53baa-dfaa-4241-9278-4449eb2f25ad" containerName="payload-pre-processing" Oct 30 03:28:11.851553 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.851513 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.859843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.859812 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.859982 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.859930 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74kz\" (UniqueName: \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.860098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.860083 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.868703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.868680 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:28:11.880959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.880936 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:28:11.883588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.883567 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z"] Oct 30 03:28:11.961315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.961279 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74kz\" (UniqueName: \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.961508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.961385 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.961508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.961432 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.962053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.962033 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.963792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.963772 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:11.980911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:11.980888 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74kz\" (UniqueName: \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\") pod \"maas-api-revoke-sub-b745e537-6bs9z\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:12.165143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:12.165069 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:12.466693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:12.466667 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z"] Oct 30 03:28:12.468317 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:28:12.468288 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5666d11b_373f_4404_9609_de88f1a72105.slice/crio-5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724 WatchSource:0}: Error finding container 5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724: Status 404 returned error can't find the container with id 5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724 Oct 30 03:28:13.001605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:13.001569 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" event={"ID":"5666d11b-373f-4404-9609-de88f1a72105","Type":"ContainerStarted","Data":"a69a5a6dfdf80fb3571edbabe533263c71b646fa7a87341ba2ae9af84ed86638"} Oct 30 03:28:13.002057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:13.001623 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" event={"ID":"5666d11b-373f-4404-9609-de88f1a72105","Type":"ContainerStarted","Data":"5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724"} Oct 30 03:28:13.041631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:13.041482 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" podStartSLOduration=2.041465742 podStartE2EDuration="2.041465742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:13.041215959 +0000 UTC m=+1521.026335848" watchObservedRunningTime="2026-10-30 03:28:13.041465742 +0000 UTC m=+1521.026585633" Oct 30 03:28:14.008438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:14.008403 4560 generic.go:356] "Generic (PLEG): container finished" podID="5666d11b-373f-4404-9609-de88f1a72105" containerID="a69a5a6dfdf80fb3571edbabe533263c71b646fa7a87341ba2ae9af84ed86638" exitCode=0 Oct 30 03:28:14.008896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:14.008484 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" event={"ID":"5666d11b-373f-4404-9609-de88f1a72105","Type":"ContainerDied","Data":"a69a5a6dfdf80fb3571edbabe533263c71b646fa7a87341ba2ae9af84ed86638"} Oct 30 03:28:15.370027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.369720 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:15.446016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.445982 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\") pod \"5666d11b-373f-4404-9609-de88f1a72105\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " Oct 30 03:28:15.446201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.446174 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\") pod \"5666d11b-373f-4404-9609-de88f1a72105\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " Oct 30 03:28:15.446300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.446285 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\" (UniqueName: \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\") pod \"5666d11b-373f-4404-9609-de88f1a72105\" (UID: \"5666d11b-373f-4404-9609-de88f1a72105\") " Oct 30 03:28:15.446415 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.446393 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca" pod "5666d11b-373f-4404-9609-de88f1a72105" (UID: "5666d11b-373f-4404-9609-de88f1a72105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:15.446813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.446786 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5666d11b-373f-4404-9609-de88f1a72105-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:15.448441 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.448410 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca" pod "5666d11b-373f-4404-9609-de88f1a72105" (UID: "5666d11b-373f-4404-9609-de88f1a72105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:15.453539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.452617 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz" pod "5666d11b-373f-4404-9609-de88f1a72105" (UID: "5666d11b-373f-4404-9609-de88f1a72105"). InnerVolumeSpecName "kube-api-access-g74kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:15.548057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.547976 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5666d11b-373f-4404-9609-de88f1a72105-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:15.548057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:15.548005 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g74kz\" (UniqueName: \"kubernetes.io/projected/5666d11b-373f-4404-9609-de88f1a72105-kube-api-access-g74kz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:16.035175 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:16.035148 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" Oct 30 03:28:16.035342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:16.035184 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z" event={"ID":"5666d11b-373f-4404-9609-de88f1a72105","Type":"ContainerDied","Data":"5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724"} Oct 30 03:28:16.035342 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:16.035255 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a881bc75afe735f960d775bb310086ce1c45d6d07f04497303ca055ba67b724" Oct 30 03:28:22.274225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.274194 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6"] Oct 30 03:28:22.274748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.274734 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5666d11b-373f-4404-9609-de88f1a72105" containerName="revoke-keys" Oct 30 03:28:22.274804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.274749 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5666d11b-373f-4404-9609-de88f1a72105" containerName="revoke-keys" Oct 30 03:28:22.274841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.274828 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5666d11b-373f-4404-9609-de88f1a72105" containerName="revoke-keys" Oct 30 03:28:22.278377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.278361 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.296844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.296817 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6"] Oct 30 03:28:22.306354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.306327 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:28:22.306540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.306411 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:28:22.332811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.332774 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.332972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.332839 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.332972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.332903 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7m9w\" (UniqueName: \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.434741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.434705 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.434938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.434754 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.434938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.434786 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7m9w\" (UniqueName: \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.435454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.435419 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.437184 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.437157 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.454895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.454865 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7m9w\" (UniqueName: \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\") pod \"maas-api-revoke-sub-369cb975-7rkd6\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.592064 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.591981 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:22.893714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:22.893680 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6"] Oct 30 03:28:22.894603 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:28:22.894578 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a48880_35b8_4793_9472_18fb9584693f.slice/crio-667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900 WatchSource:0}: Error finding container 667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900: Status 404 returned error can't find the container with id 667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900 Oct 30 03:28:23.074715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:23.074682 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" event={"ID":"39a48880-35b8-4793-9472-18fb9584693f","Type":"ContainerStarted","Data":"667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900"} Oct 30 03:28:24.082002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:24.081968 4560 generic.go:356] "Generic (PLEG): container finished" podID="39a48880-35b8-4793-9472-18fb9584693f" containerID="780263b443463821c78b95bde9f6758b92ace327d07ac1c90e41bd2e6d823abc" exitCode=0 Oct 30 03:28:24.082414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:24.082057 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" event={"ID":"39a48880-35b8-4793-9472-18fb9584693f","Type":"ContainerDied","Data":"780263b443463821c78b95bde9f6758b92ace327d07ac1c90e41bd2e6d823abc"} Oct 30 03:28:25.229277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.229254 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:25.376736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.376658 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\") pod \"39a48880-35b8-4793-9472-18fb9584693f\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " Oct 30 03:28:25.376870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.376760 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\") pod \"39a48880-35b8-4793-9472-18fb9584693f\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " Oct 30 03:28:25.377014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.376995 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\" (UniqueName: \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\") pod \"39a48880-35b8-4793-9472-18fb9584693f\" (UID: \"39a48880-35b8-4793-9472-18fb9584693f\") " Oct 30 03:28:25.377406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.377128 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca" pod "39a48880-35b8-4793-9472-18fb9584693f" (UID: "39a48880-35b8-4793-9472-18fb9584693f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:25.377406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.377366 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a48880-35b8-4793-9472-18fb9584693f-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:25.378933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.378904 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca" pod "39a48880-35b8-4793-9472-18fb9584693f" (UID: "39a48880-35b8-4793-9472-18fb9584693f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:25.379037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.378928 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w" pod "39a48880-35b8-4793-9472-18fb9584693f" (UID: "39a48880-35b8-4793-9472-18fb9584693f"). InnerVolumeSpecName "kube-api-access-r7m9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:25.478960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.478903 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7m9w\" (UniqueName: \"kubernetes.io/projected/39a48880-35b8-4793-9472-18fb9584693f-kube-api-access-r7m9w\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:25.478960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:25.478962 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a48880-35b8-4793-9472-18fb9584693f-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:26.095797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:26.095766 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" Oct 30 03:28:26.095974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:26.095797 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6" event={"ID":"39a48880-35b8-4793-9472-18fb9584693f","Type":"ContainerDied","Data":"667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900"} Oct 30 03:28:26.095974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:26.095869 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667d176bcb4707f3cc2be2efe3d5af885ca05f9a28f81abde29f5b08fe668900" Oct 30 03:28:27.827074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.827042 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:28:27.827642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.827624 4560 cpu_manager.go:414] "containerMap: removing container" podUID="39a48880-35b8-4793-9472-18fb9584693f" containerName="revoke-keys" Oct 30 03:28:27.827642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.827641 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a48880-35b8-4793-9472-18fb9584693f" containerName="revoke-keys" Oct 30 03:28:27.827763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.827723 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a48880-35b8-4793-9472-18fb9584693f" containerName="revoke-keys" Oct 30 03:28:27.831328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.831304 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.851736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.851667 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:28:27.859812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.859782 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-194506\"/\"openshift-service-ca.crt\"" Oct 30 03:28:27.859951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.859815 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-194506\"/\"kube-root-ca.crt\"" Oct 30 03:28:27.862754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.862733 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-194506\"/\"default-dockercfg-xl89h\"" Oct 30 03:28:27.862882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.862869 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-194506\"/\"praxis-route-194506-kserve-self-signed-certs\"" Oct 30 03:28:27.927608 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.927567 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.927958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.927935 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.928190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.928174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.928427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.928410 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.928609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.928594 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:27.928765 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:27.928749 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxgk\" (UniqueName: \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030281 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030325 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030348 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxgk\" (UniqueName: \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030491 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030537 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030568 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.030975 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.030949 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.031093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.031063 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.031390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.031367 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.032954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.032931 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.033171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.033148 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.053768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.053735 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxgk\" (UniqueName: \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\") pod \"praxis-route-194506-kserve-8464b8bcf9-gtspl\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.146231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.146098 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:28.460996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:28.460961 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:28:28.462782 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:28:28.462754 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedfd7049_8d47_42e4_89d6_dd02c89d2665.slice/crio-a84024436c05384d5a58631d644c7af8c4ed616b5fec56bbb908fe285fe66dfc WatchSource:0}: Error finding container a84024436c05384d5a58631d644c7af8c4ed616b5fec56bbb908fe285fe66dfc: Status 404 returned error can't find the container with id a84024436c05384d5a58631d644c7af8c4ed616b5fec56bbb908fe285fe66dfc Oct 30 03:28:29.118129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:29.118095 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" event={"ID":"edfd7049-8d47-42e4-89d6-dd02c89d2665","Type":"ContainerStarted","Data":"0f34a60e3288e51c8040c9a4e2da8989260eb860c93fd7f0c0904336ae5f38ea"} Oct 30 03:28:29.118129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:29.118135 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" event={"ID":"edfd7049-8d47-42e4-89d6-dd02c89d2665","Type":"ContainerStarted","Data":"a84024436c05384d5a58631d644c7af8c4ed616b5fec56bbb908fe285fe66dfc"} Oct 30 03:28:29.146413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:29.146383 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:38.152257 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:38.152229 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:38.159881 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:38.159855 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:38.181032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:38.181007 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:28:38.203298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:38.203246 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" podStartSLOduration=11.203234322 podStartE2EDuration="11.203234322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:29.173564411 +0000 UTC m=+1537.158684299" watchObservedRunningTime="2026-10-30 03:28:38.203234322 +0000 UTC m=+1546.188354216" Oct 30 03:28:41.469551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:41.469494 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:28:41.470654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:41.470621 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" containerID="cri-o://61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519" gracePeriod=30 Oct 30 03:28:41.749681 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:28:41.749599 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:28:46.761940 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.761910 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:28:46.863488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863454 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863509 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863638 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863670 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863725 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863749 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:28:46.863873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863770 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863848 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.863873 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863846 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:28:46.864148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863878 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.864148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.863930 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\") pod \"35c78512-53c3-4c4a-905d-147565cda2c1\" (UID: \"35c78512-53c3-4c4a-905d-147565cda2c1\") " Oct 30 03:28:46.864286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.864250 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:28:46.864495 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.864412 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.864617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.864546 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.864617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.864465 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:28:46.864713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.864626 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:46.866181 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.866157 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:28:46.866358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.866335 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "kube-api-access-kt8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:46.866437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.866413 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:28:46.866683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.866658 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token" pod "35c78512-53c3-4c4a-905d-147565cda2c1" (UID: "35c78512-53c3-4c4a-905d-147565cda2c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:46.965941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965902 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.965941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965933 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c78512-53c3-4c4a-905d-147565cda2c1-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.965941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965944 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.965941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965952 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8xz\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-kube-api-access-kt8xz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.966309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965962 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c78512-53c3-4c4a-905d-147565cda2c1-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.966309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965971 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c78512-53c3-4c4a-905d-147565cda2c1-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:46.966309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:46.965978 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c78512-53c3-4c4a-905d-147565cda2c1-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:28:47.229008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.228975 4560 generic.go:356] "Generic (PLEG): container finished" podID="35c78512-53c3-4c4a-905d-147565cda2c1" containerID="61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519" exitCode=0 Oct 30 03:28:47.229209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.229022 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" Oct 30 03:28:47.229209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.229058 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerDied","Data":"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519"} Oct 30 03:28:47.229209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.229118 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft" event={"ID":"35c78512-53c3-4c4a-905d-147565cda2c1","Type":"ContainerDied","Data":"098ae42188e96d75a4c1c47f933e5fcbaa7d02cfd312ee896b23a88c2cb15b06"} Oct 30 03:28:47.229209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.229142 4560 scope.go:122] "RemoveContainer" containerID="61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519" Oct 30 03:28:47.240130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.240114 4560 scope.go:122] "RemoveContainer" containerID="4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78" Oct 30 03:28:47.250266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.250244 4560 scope.go:122] "RemoveContainer" containerID="61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519" Oct 30 03:28:47.250570 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:28:47.250550 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519\": container with ID starting with 61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519 not found: ID does not exist" containerID="61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519" Oct 30 03:28:47.250656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.250583 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519"} err="failed to get container status \"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519\": rpc error: code = NotFound desc = could not find container \"61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519\": container with ID starting with 61bee4eed3ef06e06b26497b4c9a1eb94e5fe1ad7a46cf36edad432f90bce519 not found: ID does not exist" Oct 30 03:28:47.250656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.250611 4560 scope.go:122] "RemoveContainer" containerID="4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78" Oct 30 03:28:47.250855 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:28:47.250836 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78\": container with ID starting with 4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78 not found: ID does not exist" containerID="4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78" Oct 30 03:28:47.250897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.250862 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78"} err="failed to get container status \"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78\": rpc error: code = NotFound desc = could not find container \"4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78\": container with ID starting with 4dadbcd5db5324b492ee5d5d507cdda677e9adf0482e7b4e7baebf77319b7c78 not found: ID does not exist" Oct 30 03:28:47.282652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.282623 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:28:47.287764 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:47.287737 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-z5tft"] Oct 30 03:28:48.153262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:48.153227 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" path="/var/lib/kubelet/pods/35c78512-53c3-4c4a-905d-147565cda2c1/volumes" Oct 30 03:28:57.880233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.880198 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:28:57.880751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.880736 4560 cpu_manager.go:414] "containerMap: removing container" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:28:57.880796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.880751 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:28:57.880862 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.880852 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:28:57.880897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.880867 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:28:57.885848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.885832 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:57.917228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:57.917194 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:28:58.008404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008414 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008499 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008568 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008605 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008629 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008677 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7zm\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008712 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.008919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.008811 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110291 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110331 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110369 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7zm\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110406 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110470 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110568 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110611 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110654 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110678 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110877 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.110999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.110936 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.111092 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.111071 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.111148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.111126 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.111200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.111161 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.112895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.112871 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.113137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.113114 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.130407 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.130342 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.130560 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.130484 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7zm\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.201485 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.201454 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:28:58.520235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.520205 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:28:58.521811 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:28:58.521782 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87174551_f9aa_4157_8122_2b6290981cfc.slice/crio-fb7655a06f4e53c1a61cc23304173287c803b204094a34fb815b456f280c7b39 WatchSource:0}: Error finding container fb7655a06f4e53c1a61cc23304173287c803b204094a34fb815b456f280c7b39: Status 404 returned error can't find the container with id fb7655a06f4e53c1a61cc23304173287c803b204094a34fb815b456f280c7b39 Oct 30 03:28:58.524226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.524197 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:28:58.524297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.524259 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:28:58.524297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:58.524288 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:28:59.301567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:59.301502 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" event={"ID":"87174551-f9aa-4157-8122-2b6290981cfc","Type":"ContainerStarted","Data":"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4"} Oct 30 03:28:59.301967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:59.301590 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" event={"ID":"87174551-f9aa-4157-8122-2b6290981cfc","Type":"ContainerStarted","Data":"fb7655a06f4e53c1a61cc23304173287c803b204094a34fb815b456f280c7b39"} Oct 30 03:28:59.376681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:28:59.376616 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podStartSLOduration=2.376597684 podStartE2EDuration="2.376597684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:59.366845659 +0000 UTC m=+1567.351965562" watchObservedRunningTime="2026-10-30 03:28:59.376597684 +0000 UTC m=+1567.361717597" Oct 30 03:29:00.201925 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:00.201890 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:29:00.208222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:00.203395 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:00.208222 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:00.203458 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:01.201971 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:01.201938 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:01.202347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:01.201991 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:02.202209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:02.202172 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:02.202643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:02.202231 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:03.202397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:03.202365 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:03.202901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:03.202433 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:03.753544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:03.747695 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:29:03.753544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:03.747928 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" containerID="cri-o://5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8" gracePeriod=30 Oct 30 03:29:04.202032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:04.202009 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:04.202179 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:04.202060 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:05.202039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:05.201997 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 03:29:05.202580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:05.202057 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 03:29:06.207432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:06.207391 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:29:06.207879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:06.207842 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:29:06.208667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:06.208647 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:29:09.014490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.014468 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:29:09.067572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067467 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067538 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067584 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067666 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067696 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067776 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067811 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.067898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067844 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.068196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067939 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:09.068196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.067968 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\") pod \"87580e59-cc07-4381-a815-e4bd720c8b68\" (UID: \"87580e59-cc07-4381-a815-e4bd720c8b68\") " Oct 30 03:29:09.068196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.068061 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:09.068196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.068186 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.068647 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.068764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.068991 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.069011 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.069026 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.069038 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87580e59-cc07-4381-a815-e4bd720c8b68-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.069959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.069050 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.070902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.070868 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:09.071110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.071085 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "kube-api-access-kxsll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:09.071596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.071572 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:29:09.071693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.071619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token" pod "87580e59-cc07-4381-a815-e4bd720c8b68" (UID: "87580e59-cc07-4381-a815-e4bd720c8b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:09.170851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.170817 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.170851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.170851 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87580e59-cc07-4381-a815-e4bd720c8b68-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.170851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.170862 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxsll\" (UniqueName: \"kubernetes.io/projected/87580e59-cc07-4381-a815-e4bd720c8b68-kube-api-access-kxsll\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.171184 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.170873 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87580e59-cc07-4381-a815-e4bd720c8b68-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:09.367895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.367812 4560 generic.go:356] "Generic (PLEG): container finished" podID="87580e59-cc07-4381-a815-e4bd720c8b68" containerID="5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8" exitCode=0 Oct 30 03:29:09.367895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.367874 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" Oct 30 03:29:09.368095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.367893 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerDied","Data":"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8"} Oct 30 03:29:09.368095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.367972 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b" event={"ID":"87580e59-cc07-4381-a815-e4bd720c8b68","Type":"ContainerDied","Data":"0c37f7bdeddf658f9f6f867b9c39d6a4f7ff12d450a40203cfe4a5430922334a"} Oct 30 03:29:09.368095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.367995 4560 scope.go:122] "RemoveContainer" containerID="5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8" Oct 30 03:29:09.378387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.378363 4560 scope.go:122] "RemoveContainer" containerID="4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd" Oct 30 03:29:09.387199 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.387176 4560 scope.go:122] "RemoveContainer" containerID="5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8" Oct 30 03:29:09.387473 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:29:09.387454 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8\": container with ID starting with 5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8 not found: ID does not exist" containerID="5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8" Oct 30 03:29:09.387539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.387482 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8"} err="failed to get container status \"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8\": rpc error: code = NotFound desc = could not find container \"5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8\": container with ID starting with 5dae878ce30c5af6672b84f626c834d197d2af325fd6ce6a97d57bafc789d5e8 not found: ID does not exist" Oct 30 03:29:09.387539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.387499 4560 scope.go:122] "RemoveContainer" containerID="4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd" Oct 30 03:29:09.387762 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:29:09.387744 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd\": container with ID starting with 4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd not found: ID does not exist" containerID="4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd" Oct 30 03:29:09.387805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.387769 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd"} err="failed to get container status \"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd\": rpc error: code = NotFound desc = could not find container \"4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd\": container with ID starting with 4aa0d4121386c6c9dea671d8fc4f5c6ef718ff05efcbbdf6bf23009c54bf0acd not found: ID does not exist" Oct 30 03:29:09.420707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.420672 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:29:09.433387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:09.433353 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-efbbc5-openshift-default-69965c4795-6wk4b"] Oct 30 03:29:10.151573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:10.151539 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" path="/var/lib/kubelet/pods/87580e59-cc07-4381-a815-e4bd720c8b68/volumes" Oct 30 03:29:11.749610 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:29:11.749563 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:29:41.749467 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:29:41.749430 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:29:44.440790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:44.440750 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:29:59.257593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.257556 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" podUID="b6ab5ae10a035469fb072566acf5cdc5" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 03:29:59.461875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.461831 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" podUID="edfd7049-8d47-42e4-89d6-dd02c89d2665" containerName="main" containerID="cri-o://0f34a60e3288e51c8040c9a4e2da8989260eb860c93fd7f0c0904336ae5f38ea" gracePeriod=45 Oct 30 03:29:59.663614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.663570 4560 generic.go:356] "Generic (PLEG): container finished" podID="edfd7049-8d47-42e4-89d6-dd02c89d2665" containerID="0f34a60e3288e51c8040c9a4e2da8989260eb860c93fd7f0c0904336ae5f38ea" exitCode=0 Oct 30 03:29:59.663803 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.663636 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" event={"ID":"edfd7049-8d47-42e4-89d6-dd02c89d2665","Type":"ContainerDied","Data":"0f34a60e3288e51c8040c9a4e2da8989260eb860c93fd7f0c0904336ae5f38ea"} Oct 30 03:29:59.726685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.726657 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:29:59.769457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769424 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\" (UniqueName: \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.769686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769481 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.769686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769612 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.769686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769665 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\" (UniqueName: \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769705 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.769949 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.770458 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\") pod \"edfd7049-8d47-42e4-89d6-dd02c89d2665\" (UID: \"edfd7049-8d47-42e4-89d6-dd02c89d2665\") " Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.770585 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.770945 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:59.771773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.770965 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:59.773411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.773368 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:59.773839 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.773817 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:59.774703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.774682 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:59.775582 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.775546 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk" pod "edfd7049-8d47-42e4-89d6-dd02c89d2665" (UID: "edfd7049-8d47-42e4-89d6-dd02c89d2665"). InnerVolumeSpecName "kube-api-access-6fxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:59.872578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.872486 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:59.872578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.872553 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxgk\" (UniqueName: \"kubernetes.io/projected/edfd7049-8d47-42e4-89d6-dd02c89d2665-kube-api-access-6fxgk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:59.872578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.872567 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfd7049-8d47-42e4-89d6-dd02c89d2665-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:29:59.872578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:29:59.872576 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edfd7049-8d47-42e4-89d6-dd02c89d2665-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.166533 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167489 4560 cpu_manager.go:414] "containerMap: removing container" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167509 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167561 4560 cpu_manager.go:414] "containerMap: removing container" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167572 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167601 4560 cpu_manager.go:414] "containerMap: removing container" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167610 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c78512-53c3-4c4a-905d-147565cda2c1" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167622 4560 cpu_manager.go:414] "containerMap: removing container" podUID="edfd7049-8d47-42e4-89d6-dd02c89d2665" containerName="main" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167629 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edfd7049-8d47-42e4-89d6-dd02c89d2665" containerName="main" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167768 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167787 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="edfd7049-8d47-42e4-89d6-dd02c89d2665" containerName="main" Oct 30 03:30:00.173918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.167798 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="87580e59-cc07-4381-a815-e4bd720c8b68" containerName="istio-proxy" Oct 30 03:30:00.176352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.176324 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:00.181569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.180212 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.183320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.183287 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:00.184534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.184165 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.194028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.193972 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:00.194028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.194017 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:00.194228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.194044 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:00.199237 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.198973 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.206785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.206751 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.208680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.208619 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:00.208947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.208928 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:30:00.212001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.211974 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:00.278140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278107 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gdb\" (UniqueName: \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.278140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278147 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.278632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278259 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvv54\" (UniqueName: \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.278632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278398 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xqs\" (UniqueName: \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.278632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278436 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.278632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278455 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.278632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278620 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.278790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.278654 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rxw\" (UniqueName: \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.380370 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380334 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.380370 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380375 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rxw\" (UniqueName: \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.380622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380399 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gdb\" (UniqueName: \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.380622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380429 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.380622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380493 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvv54\" (UniqueName: \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.380796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380640 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xqs\" (UniqueName: \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.380796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380699 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.380796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.380739 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.381240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.381217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.381277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.381217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.381311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.381285 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.381358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.381340 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.400837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.400805 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rxw\" (UniqueName: \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\") pod \"maas-api-key-cleanup-29888850-jhjsp\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.400966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.400881 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gdb\" (UniqueName: \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\") pod \"maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.401079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.401058 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvv54\" (UniqueName: \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.401115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.401059 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xqs\" (UniqueName: \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\") pod \"maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.517215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.517172 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:00.541587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.541548 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:00.552863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.552834 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:00.562512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.562483 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:00.676015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.674982 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" event={"ID":"edfd7049-8d47-42e4-89d6-dd02c89d2665","Type":"ContainerDied","Data":"a84024436c05384d5a58631d644c7af8c4ed616b5fec56bbb908fe285fe66dfc"} Oct 30 03:30:00.676015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.675027 4560 scope.go:122] "RemoveContainer" containerID="0f34a60e3288e51c8040c9a4e2da8989260eb860c93fd7f0c0904336ae5f38ea" Oct 30 03:30:00.676015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.675153 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl" Oct 30 03:30:00.729615 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.729575 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:30:00.731968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.731941 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-194506/praxis-route-194506-kserve-8464b8bcf9-gtspl"] Oct 30 03:30:00.826345 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.826313 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:00.827144 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:30:00.827119 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769aea49_1206_430d_9963_a780b46b2ae9.slice/crio-b0ba5fb5ac58403c73cec3b0313ad680ca3891238ce64e489df858903f4d1169 WatchSource:0}: Error finding container b0ba5fb5ac58403c73cec3b0313ad680ca3891238ce64e489df858903f4d1169: Status 404 returned error can't find the container with id b0ba5fb5ac58403c73cec3b0313ad680ca3891238ce64e489df858903f4d1169 Oct 30 03:30:00.872691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.872657 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:00.873638 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:30:00.873614 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947cb936_ce35_431d_ac15_9116c736fda5.slice/crio-1bd6b367e8a601498dc0c68eb75a2951b416e5a1e2cd60195a85e0a6a16dbbc7 WatchSource:0}: Error finding container 1bd6b367e8a601498dc0c68eb75a2951b416e5a1e2cd60195a85e0a6a16dbbc7: Status 404 returned error can't find the container with id 1bd6b367e8a601498dc0c68eb75a2951b416e5a1e2cd60195a85e0a6a16dbbc7 Oct 30 03:30:00.906304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.906274 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:00.907640 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:30:00.907608 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c445d_7d72_4e1f_9289_760d7353a9b1.slice/crio-2314c0ed06864ff19a772afff297e001ddf40c7610360d0e248ebf43ce76e719 WatchSource:0}: Error finding container 2314c0ed06864ff19a772afff297e001ddf40c7610360d0e248ebf43ce76e719: Status 404 returned error can't find the container with id 2314c0ed06864ff19a772afff297e001ddf40c7610360d0e248ebf43ce76e719 Oct 30 03:30:00.910234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:00.910212 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:00.910702 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:30:00.910681 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac0cd33_54bb_44e6_b25d_1d3ce784669e.slice/crio-bdbfa05e809b941033626c6a0ba9403a9dd89fa284d3cdc6d782ac4a7c811e05 WatchSource:0}: Error finding container bdbfa05e809b941033626c6a0ba9403a9dd89fa284d3cdc6d782ac4a7c811e05: Status 404 returned error can't find the container with id bdbfa05e809b941033626c6a0ba9403a9dd89fa284d3cdc6d782ac4a7c811e05 Oct 30 03:30:01.683098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.683006 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerStarted","Data":"a10a027d7c10ea689ed26fcb8b0d41634904dcb8476d0886d4afd75dda5a77dd"} Oct 30 03:30:01.683098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.683076 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerStarted","Data":"bdbfa05e809b941033626c6a0ba9403a9dd89fa284d3cdc6d782ac4a7c811e05"} Oct 30 03:30:01.684812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.684786 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerStarted","Data":"dfc6bfca573e7976f9c83b57fa3f5a3ec590259967501a90bc1d50547fc4f596"} Oct 30 03:30:01.684937 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.684834 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerStarted","Data":"1bd6b367e8a601498dc0c68eb75a2951b416e5a1e2cd60195a85e0a6a16dbbc7"} Oct 30 03:30:01.686438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.686409 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerStarted","Data":"4080bf7df75cc81849d52bcd694243803cdfbd47041b5124101aa6d8796ee20d"} Oct 30 03:30:01.686587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.686450 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerStarted","Data":"b0ba5fb5ac58403c73cec3b0313ad680ca3891238ce64e489df858903f4d1169"} Oct 30 03:30:01.690782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.689967 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerStarted","Data":"11cb6781f905cb8874d1859f3c0428a8077a03ec6e163bd542720a9d2b361a1e"} Oct 30 03:30:01.690782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.690068 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerStarted","Data":"2314c0ed06864ff19a772afff297e001ddf40c7610360d0e248ebf43ce76e719"} Oct 30 03:30:01.722121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.722057 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" podStartSLOduration=1.722041587 podStartE2EDuration="1.722041587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.721893383 +0000 UTC m=+1629.707013275" watchObservedRunningTime="2026-10-30 03:30:01.722041587 +0000 UTC m=+1629.707161475" Oct 30 03:30:01.762531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.762393 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" podStartSLOduration=1.7623790000000001 podStartE2EDuration="1.762379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.760483258 +0000 UTC m=+1629.745603140" watchObservedRunningTime="2026-10-30 03:30:01.762379 +0000 UTC m=+1629.747498915" Oct 30 03:30:01.801218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.801165 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" podStartSLOduration=1.801151499 podStartE2EDuration="1.801151499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.798596839 +0000 UTC m=+1629.783716737" watchObservedRunningTime="2026-10-30 03:30:01.801151499 +0000 UTC m=+1629.786271399" Oct 30 03:30:01.836896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:01.836843 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" podStartSLOduration=1.8368282649999998 podStartE2EDuration="1.836828265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.835973165 +0000 UTC m=+1629.821093158" watchObservedRunningTime="2026-10-30 03:30:01.836828265 +0000 UTC m=+1629.821948176" Oct 30 03:30:02.157403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.157362 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edfd7049-8d47-42e4-89d6-dd02c89d2665" path="/var/lib/kubelet/pods/edfd7049-8d47-42e4-89d6-dd02c89d2665/volumes" Oct 30 03:30:02.697543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.697492 4560 generic.go:356] "Generic (PLEG): container finished" podID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerID="a10a027d7c10ea689ed26fcb8b0d41634904dcb8476d0886d4afd75dda5a77dd" exitCode=60 Oct 30 03:30:02.698012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.697583 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerDied","Data":"a10a027d7c10ea689ed26fcb8b0d41634904dcb8476d0886d4afd75dda5a77dd"} Oct 30 03:30:02.698012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.697919 4560 scope.go:122] "RemoveContainer" containerID="a10a027d7c10ea689ed26fcb8b0d41634904dcb8476d0886d4afd75dda5a77dd" Oct 30 03:30:02.699289 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.699252 4560 generic.go:356] "Generic (PLEG): container finished" podID="947cb936-ce35-431d-ac15-9116c736fda5" containerID="dfc6bfca573e7976f9c83b57fa3f5a3ec590259967501a90bc1d50547fc4f596" exitCode=60 Oct 30 03:30:02.699410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.699343 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerDied","Data":"dfc6bfca573e7976f9c83b57fa3f5a3ec590259967501a90bc1d50547fc4f596"} Oct 30 03:30:02.699704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.699601 4560 scope.go:122] "RemoveContainer" containerID="dfc6bfca573e7976f9c83b57fa3f5a3ec590259967501a90bc1d50547fc4f596" Oct 30 03:30:02.701051 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.701027 4560 generic.go:356] "Generic (PLEG): container finished" podID="769aea49-1206-430d-9963-a780b46b2ae9" containerID="4080bf7df75cc81849d52bcd694243803cdfbd47041b5124101aa6d8796ee20d" exitCode=60 Oct 30 03:30:02.701119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.701054 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerDied","Data":"4080bf7df75cc81849d52bcd694243803cdfbd47041b5124101aa6d8796ee20d"} Oct 30 03:30:02.701463 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.701437 4560 scope.go:122] "RemoveContainer" containerID="4080bf7df75cc81849d52bcd694243803cdfbd47041b5124101aa6d8796ee20d" Oct 30 03:30:02.707347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.707322 4560 generic.go:356] "Generic (PLEG): container finished" podID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerID="11cb6781f905cb8874d1859f3c0428a8077a03ec6e163bd542720a9d2b361a1e" exitCode=60 Oct 30 03:30:02.707457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.707438 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerDied","Data":"11cb6781f905cb8874d1859f3c0428a8077a03ec6e163bd542720a9d2b361a1e"} Oct 30 03:30:02.707634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:02.707620 4560 scope.go:122] "RemoveContainer" containerID="11cb6781f905cb8874d1859f3c0428a8077a03ec6e163bd542720a9d2b361a1e" Oct 30 03:30:03.715330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:03.715286 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerStarted","Data":"9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361"} Oct 30 03:30:03.717120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:03.717093 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerStarted","Data":"8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c"} Oct 30 03:30:03.718985 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:03.718960 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerStarted","Data":"e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794"} Oct 30 03:30:03.720900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:03.720875 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerStarted","Data":"c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445"} Oct 30 03:30:04.728468 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.728431 4560 generic.go:356] "Generic (PLEG): container finished" podID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerID="9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361" exitCode=60 Oct 30 03:30:04.729107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.728531 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerDied","Data":"9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361"} Oct 30 03:30:04.729107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.728604 4560 scope.go:122] "RemoveContainer" containerID="a10a027d7c10ea689ed26fcb8b0d41634904dcb8476d0886d4afd75dda5a77dd" Oct 30 03:30:04.729107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.728790 4560 scope.go:122] "RemoveContainer" containerID="9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361" Oct 30 03:30:04.729282 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:04.729107 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46_odh-ai-gateway-infra(8ac0cd33-54bb-44e6-b25d-1d3ce784669e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" Oct 30 03:30:04.730419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.730402 4560 generic.go:356] "Generic (PLEG): container finished" podID="947cb936-ce35-431d-ac15-9116c736fda5" containerID="8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c" exitCode=60 Oct 30 03:30:04.730512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.730463 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerDied","Data":"8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c"} Oct 30 03:30:04.730754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.730739 4560 scope.go:122] "RemoveContainer" containerID="8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c" Oct 30 03:30:04.731005 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:04.730986 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c_odh-ai-gateway-infra(947cb936-ce35-431d-ac15-9116c736fda5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" podUID="947cb936-ce35-431d-ac15-9116c736fda5" Oct 30 03:30:04.732277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.732247 4560 generic.go:356] "Generic (PLEG): container finished" podID="769aea49-1206-430d-9963-a780b46b2ae9" containerID="e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794" exitCode=60 Oct 30 03:30:04.732381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.732345 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerDied","Data":"e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794"} Oct 30 03:30:04.732605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.732588 4560 scope.go:122] "RemoveContainer" containerID="e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794" Oct 30 03:30:04.732826 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:04.732809 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-jhjsp_odh-ai-gateway-infra(769aea49-1206-430d-9963-a780b46b2ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" podUID="769aea49-1206-430d-9963-a780b46b2ae9" Oct 30 03:30:04.734379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.734356 4560 generic.go:356] "Generic (PLEG): container finished" podID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerID="c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445" exitCode=60 Oct 30 03:30:04.734460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.734430 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerDied","Data":"c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445"} Oct 30 03:30:04.734597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.734582 4560 scope.go:122] "RemoveContainer" containerID="c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445" Oct 30 03:30:04.734765 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:04.734749 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7_odh-ai-gateway-infra(0a5c445d-7d72-4e1f-9289-760d7353a9b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" Oct 30 03:30:04.740475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.740458 4560 scope.go:122] "RemoveContainer" containerID="dfc6bfca573e7976f9c83b57fa3f5a3ec590259967501a90bc1d50547fc4f596" Oct 30 03:30:04.752726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.752704 4560 scope.go:122] "RemoveContainer" containerID="4080bf7df75cc81849d52bcd694243803cdfbd47041b5124101aa6d8796ee20d" Oct 30 03:30:04.765350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.765332 4560 scope.go:122] "RemoveContainer" containerID="11cb6781f905cb8874d1859f3c0428a8077a03ec6e163bd542720a9d2b361a1e" Oct 30 03:30:04.970884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.970852 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:30:04.976094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.976070 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:04.994278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.994164 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-773fbb-serving-cert\"" Oct 30 03:30:04.995215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:04.995186 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:30:05.036670 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.036635 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.036867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.036763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.036867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.036816 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.037694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.037669 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spv2n\" (UniqueName: \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.138964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.138918 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.139151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.139016 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.139151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.139057 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.139151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.139148 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spv2n\" (UniqueName: \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.139755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.139729 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.141493 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.141469 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.141627 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.141532 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.158699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.158675 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spv2n\" (UniqueName: \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\") pod \"maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.298319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.298232 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:05.603980 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.603948 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:30:05.604820 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:30:05.604783 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5527d7f6_4880_4bb5_961d_584f8af08d5a.slice/crio-dac82767cc2b637787273ec0051b4017ac6a2a7841db639448f6673355145c35 WatchSource:0}: Error finding container dac82767cc2b637787273ec0051b4017ac6a2a7841db639448f6673355145c35: Status 404 returned error can't find the container with id dac82767cc2b637787273ec0051b4017ac6a2a7841db639448f6673355145c35 Oct 30 03:30:05.742566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.742510 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" event={"ID":"5527d7f6-4880-4bb5-961d-584f8af08d5a","Type":"ContainerStarted","Data":"dac82767cc2b637787273ec0051b4017ac6a2a7841db639448f6673355145c35"} Oct 30 03:30:05.744802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.744779 4560 scope.go:122] "RemoveContainer" containerID="9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361" Oct 30 03:30:05.745111 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:05.745089 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46_odh-ai-gateway-infra(8ac0cd33-54bb-44e6-b25d-1d3ce784669e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" Oct 30 03:30:05.746886 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.746861 4560 scope.go:122] "RemoveContainer" containerID="8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c" Oct 30 03:30:05.747135 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:05.747108 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c_odh-ai-gateway-infra(947cb936-ce35-431d-ac15-9116c736fda5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" podUID="947cb936-ce35-431d-ac15-9116c736fda5" Oct 30 03:30:05.748437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.748414 4560 scope.go:122] "RemoveContainer" containerID="e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794" Oct 30 03:30:05.748658 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:05.748640 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-jhjsp_odh-ai-gateway-infra(769aea49-1206-430d-9963-a780b46b2ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" podUID="769aea49-1206-430d-9963-a780b46b2ae9" Oct 30 03:30:05.749946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:05.749927 4560 scope.go:122] "RemoveContainer" containerID="c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445" Oct 30 03:30:05.750153 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:05.750134 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7_odh-ai-gateway-infra(0a5c445d-7d72-4e1f-9289-760d7353a9b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" Oct 30 03:30:06.757578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:06.757543 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" event={"ID":"5527d7f6-4880-4bb5-961d-584f8af08d5a","Type":"ContainerStarted","Data":"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c"} Oct 30 03:30:06.758053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:06.757626 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:06.800347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:06.800297 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" podStartSLOduration=2.454602052 podStartE2EDuration="2.800281814s" totalImagesPullingTime="345.679762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:06.797977652 +0000 UTC m=+1634.783097541" watchObservedRunningTime="2026-10-30 03:30:06.800281814 +0000 UTC m=+1634.785401721" Oct 30 03:30:11.749185 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:11.749148 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:30:12.769303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:12.769276 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:30:15.145718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:15.145681 4560 scope.go:122] "RemoveContainer" containerID="8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c" Oct 30 03:30:15.816387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:15.816307 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerStarted","Data":"84ccc10ae6d50804a53c815bf410b0b99086aaec573a5ecdf5956ec574f26ba9"} Oct 30 03:30:16.823295 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:16.823256 4560 generic.go:356] "Generic (PLEG): container finished" podID="947cb936-ce35-431d-ac15-9116c736fda5" containerID="84ccc10ae6d50804a53c815bf410b0b99086aaec573a5ecdf5956ec574f26ba9" exitCode=60 Oct 30 03:30:16.823811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:16.823328 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerDied","Data":"84ccc10ae6d50804a53c815bf410b0b99086aaec573a5ecdf5956ec574f26ba9"} Oct 30 03:30:16.823811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:16.823398 4560 scope.go:122] "RemoveContainer" containerID="8407e73b90c053e580bf6459d89afb0c633556295b6a06480ff563ae4aea680c" Oct 30 03:30:16.823811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:16.823788 4560 scope.go:122] "RemoveContainer" containerID="84ccc10ae6d50804a53c815bf410b0b99086aaec573a5ecdf5956ec574f26ba9" Oct 30 03:30:16.824102 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:16.824078 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c_odh-ai-gateway-infra(947cb936-ce35-431d-ac15-9116c736fda5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" podUID="947cb936-ce35-431d-ac15-9116c736fda5" Oct 30 03:30:16.871901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:16.871871 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:17.145809 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:17.145731 4560 scope.go:122] "RemoveContainer" containerID="9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361" Oct 30 03:30:17.832114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:17.832024 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerStarted","Data":"7927c521d560ec9877a29736f32c6dd9233b02e0823913c8cdb5d3f223f82933"} Oct 30 03:30:18.042938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.042912 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:18.156240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.156141 4560 scope.go:122] "RemoveContainer" containerID="c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445" Oct 30 03:30:18.156431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.156381 4560 scope.go:122] "RemoveContainer" containerID="e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794" Oct 30 03:30:18.217081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.217055 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\") pod \"947cb936-ce35-431d-ac15-9116c736fda5\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " Oct 30 03:30:18.217205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.217178 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\" (UniqueName: \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\") pod \"947cb936-ce35-431d-ac15-9116c736fda5\" (UID: \"947cb936-ce35-431d-ac15-9116c736fda5\") " Oct 30 03:30:18.217396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.217372 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca" pod "947cb936-ce35-431d-ac15-9116c736fda5" (UID: "947cb936-ce35-431d-ac15-9116c736fda5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:18.217593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.217579 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947cb936-ce35-431d-ac15-9116c736fda5-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:18.219140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.219122 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb" pod "947cb936-ce35-431d-ac15-9116c736fda5" (UID: "947cb936-ce35-431d-ac15-9116c736fda5"). InnerVolumeSpecName "kube-api-access-k8gdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:18.319027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.318987 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8gdb\" (UniqueName: \"kubernetes.io/projected/947cb936-ce35-431d-ac15-9116c736fda5-kube-api-access-k8gdb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:18.841676 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.841589 4560 generic.go:356] "Generic (PLEG): container finished" podID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerID="7927c521d560ec9877a29736f32c6dd9233b02e0823913c8cdb5d3f223f82933" exitCode=60 Oct 30 03:30:18.842118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.841666 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerDied","Data":"7927c521d560ec9877a29736f32c6dd9233b02e0823913c8cdb5d3f223f82933"} Oct 30 03:30:18.842118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.841733 4560 scope.go:122] "RemoveContainer" containerID="9eab91eb29e2f93c07a889a24ce2f53115ed36773219cbe4900d66fcbfe0e361" Oct 30 03:30:18.842118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.842067 4560 scope.go:122] "RemoveContainer" containerID="7927c521d560ec9877a29736f32c6dd9233b02e0823913c8cdb5d3f223f82933" Oct 30 03:30:18.842372 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:18.842349 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46_odh-ai-gateway-infra(8ac0cd33-54bb-44e6-b25d-1d3ce784669e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" Oct 30 03:30:18.843008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.842989 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" event={"ID":"947cb936-ce35-431d-ac15-9116c736fda5","Type":"ContainerDied","Data":"1bd6b367e8a601498dc0c68eb75a2951b416e5a1e2cd60195a85e0a6a16dbbc7"} Oct 30 03:30:18.843068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.842994 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c" Oct 30 03:30:18.844974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.844951 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerStarted","Data":"da3fe7e9fcc4f5e3a0240a45c003a6684f75b70a8147ad07157b76b7465a6162"} Oct 30 03:30:18.846895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.846872 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerStarted","Data":"ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca"} Oct 30 03:30:18.857662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.857492 4560 scope.go:122] "RemoveContainer" containerID="84ccc10ae6d50804a53c815bf410b0b99086aaec573a5ecdf5956ec574f26ba9" Oct 30 03:30:18.883983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.883949 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:18.932364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.932335 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:18.941571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:18.941546 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-65f21d-29888850-t7x5c"] Oct 30 03:30:19.002953 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:19.002917 4560 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c445d_7d72_4e1f_9289_760d7353a9b1.slice/crio-ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c445d_7d72_4e1f_9289_760d7353a9b1.slice/crio-conmon-ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca.scope\": RecentStats: unable to find data in memory cache]" Oct 30 03:30:19.863900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.863870 4560 generic.go:356] "Generic (PLEG): container finished" podID="769aea49-1206-430d-9963-a780b46b2ae9" containerID="da3fe7e9fcc4f5e3a0240a45c003a6684f75b70a8147ad07157b76b7465a6162" exitCode=60 Oct 30 03:30:19.864267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.863940 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerDied","Data":"da3fe7e9fcc4f5e3a0240a45c003a6684f75b70a8147ad07157b76b7465a6162"} Oct 30 03:30:19.864267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.864044 4560 scope.go:122] "RemoveContainer" containerID="e36722515df3ec19d6255e04c8d6ebf75e264575341ed6a866a0eb12cb1fa794" Oct 30 03:30:19.864390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.864373 4560 scope.go:122] "RemoveContainer" containerID="da3fe7e9fcc4f5e3a0240a45c003a6684f75b70a8147ad07157b76b7465a6162" Oct 30 03:30:19.864696 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:19.864663 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-jhjsp_odh-ai-gateway-infra(769aea49-1206-430d-9963-a780b46b2ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" podUID="769aea49-1206-430d-9963-a780b46b2ae9" Oct 30 03:30:19.865912 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.865889 4560 generic.go:356] "Generic (PLEG): container finished" podID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerID="ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca" exitCode=60 Oct 30 03:30:19.865990 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.865959 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerDied","Data":"ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca"} Oct 30 03:30:19.866251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.866237 4560 scope.go:122] "RemoveContainer" containerID="ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca" Oct 30 03:30:19.866476 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:19.866459 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7_odh-ai-gateway-infra(0a5c445d-7d72-4e1f-9289-760d7353a9b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" Oct 30 03:30:19.877170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.877144 4560 scope.go:122] "RemoveContainer" containerID="c689de0804a7b1e6bc91064f112c111682b541379205362d37df75ca45a09445" Oct 30 03:30:19.988037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:19.988006 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:20.020339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.020304 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:20.026388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.026360 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:20.037556 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.037513 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\" (UniqueName: \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\") pod \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " Oct 30 03:30:20.037699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.037678 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\") pod \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\" (UID: \"8ac0cd33-54bb-44e6-b25d-1d3ce784669e\") " Oct 30 03:30:20.038070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.038045 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca" pod "8ac0cd33-54bb-44e6-b25d-1d3ce784669e" (UID: "8ac0cd33-54bb-44e6-b25d-1d3ce784669e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:20.038189 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.038150 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:20.039942 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.039916 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54" pod "8ac0cd33-54bb-44e6-b25d-1d3ce784669e" (UID: "8ac0cd33-54bb-44e6-b25d-1d3ce784669e"). InnerVolumeSpecName "kube-api-access-fvv54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:20.147743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.139715 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvv54\" (UniqueName: \"kubernetes.io/projected/8ac0cd33-54bb-44e6-b25d-1d3ce784669e-kube-api-access-fvv54\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:20.156716 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.156682 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947cb936-ce35-431d-ac15-9116c736fda5" path="/var/lib/kubelet/pods/947cb936-ce35-431d-ac15-9116c736fda5/volumes" Oct 30 03:30:20.562098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.562062 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:30:20.596409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.596379 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:30:20.873044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.872964 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" Oct 30 03:30:20.873044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.872961 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46" event={"ID":"8ac0cd33-54bb-44e6-b25d-1d3ce784669e","Type":"ContainerDied","Data":"bdbfa05e809b941033626c6a0ba9403a9dd89fa284d3cdc6d782ac4a7c811e05"} Oct 30 03:30:20.873614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.873085 4560 scope.go:122] "RemoveContainer" containerID="7927c521d560ec9877a29736f32c6dd9233b02e0823913c8cdb5d3f223f82933" Oct 30 03:30:20.931435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.931388 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:20.936795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:20.936759 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888850-r6k46"] Oct 30 03:30:21.082609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.082583 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:21.084846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.084828 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:21.150958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.150864 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\" (UniqueName: \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\") pod \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " Oct 30 03:30:21.151136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.150960 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\") pod \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\" (UID: \"0a5c445d-7d72-4e1f-9289-760d7353a9b1\") " Oct 30 03:30:21.151136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151049 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\") pod \"769aea49-1206-430d-9963-a780b46b2ae9\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " Oct 30 03:30:21.151136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151128 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\" (UniqueName: \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\") pod \"769aea49-1206-430d-9963-a780b46b2ae9\" (UID: \"769aea49-1206-430d-9963-a780b46b2ae9\") " Oct 30 03:30:21.151409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151373 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca" pod "0a5c445d-7d72-4e1f-9289-760d7353a9b1" (UID: "0a5c445d-7d72-4e1f-9289-760d7353a9b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:21.151499 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151412 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca" pod "769aea49-1206-430d-9963-a780b46b2ae9" (UID: "769aea49-1206-430d-9963-a780b46b2ae9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:21.151639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151621 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769aea49-1206-430d-9963-a780b46b2ae9-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.151705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.151645 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c445d-7d72-4e1f-9289-760d7353a9b1-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.156552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.154009 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw" pod "769aea49-1206-430d-9963-a780b46b2ae9" (UID: "769aea49-1206-430d-9963-a780b46b2ae9"). InnerVolumeSpecName "kube-api-access-27rxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:21.156552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.154322 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs" pod "0a5c445d-7d72-4e1f-9289-760d7353a9b1" (UID: "0a5c445d-7d72-4e1f-9289-760d7353a9b1"). InnerVolumeSpecName "kube-api-access-t2xqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:21.254887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.254847 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rxw\" (UniqueName: \"kubernetes.io/projected/769aea49-1206-430d-9963-a780b46b2ae9-kube-api-access-27rxw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.254887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.254887 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xqs\" (UniqueName: \"kubernetes.io/projected/0a5c445d-7d72-4e1f-9289-760d7353a9b1-kube-api-access-t2xqs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.885642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.885606 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" Oct 30 03:30:21.886104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.885609 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp" event={"ID":"769aea49-1206-430d-9963-a780b46b2ae9","Type":"ContainerDied","Data":"b0ba5fb5ac58403c73cec3b0313ad680ca3891238ce64e489df858903f4d1169"} Oct 30 03:30:21.886104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.885729 4560 scope.go:122] "RemoveContainer" containerID="da3fe7e9fcc4f5e3a0240a45c003a6684f75b70a8147ad07157b76b7465a6162" Oct 30 03:30:21.887026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.887008 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" Oct 30 03:30:21.887122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.887062 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7" event={"ID":"0a5c445d-7d72-4e1f-9289-760d7353a9b1","Type":"ContainerDied","Data":"2314c0ed06864ff19a772afff297e001ddf40c7610360d0e248ebf43ce76e719"} Oct 30 03:30:21.895050 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.895029 4560 scope.go:122] "RemoveContainer" containerID="ac662294c394ce0879b562716f9c909cc7b7fa4df7087fa454abd75e7a3f13ca" Oct 30 03:30:21.940132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.940100 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:21.952795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.952767 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-jhjsp"] Oct 30 03:30:21.981291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.981261 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:21.989001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:21.988977 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-194506-29888850-vd9p7"] Oct 30 03:30:22.155574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:22.155471 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" path="/var/lib/kubelet/pods/0a5c445d-7d72-4e1f-9289-760d7353a9b1/volumes" Oct 30 03:30:22.155995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:22.155966 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769aea49-1206-430d-9963-a780b46b2ae9" path="/var/lib/kubelet/pods/769aea49-1206-430d-9963-a780b46b2ae9/volumes" Oct 30 03:30:22.156389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:22.156372 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" path="/var/lib/kubelet/pods/8ac0cd33-54bb-44e6-b25d-1d3ce784669e/volumes" Oct 30 03:30:25.563000 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:25.562920 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" containerID="cri-o://14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e" gracePeriod=25 Oct 30 03:30:25.597451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:25.597408 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" containerID="cri-o://763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83" gracePeriod=25 Oct 30 03:30:34.463825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:34.463719 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:30:34.463825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:34.463718 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:30:35.507593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.507558 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:30:35.509602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.509566 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" podUID="6e43c67c-693c-4c85-84ed-0f38542ea858" containerName="maas-api" containerID="cri-o://ea6ea71a3a975d5bc8ab23ebae4ed87a21e6e7f6bfca0ae2876773a3b8f78d73" gracePeriod=30 Oct 30 03:30:35.561739 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.561700 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:30:35.562007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.561982 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" containerID="cri-o://ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0" gracePeriod=30 Oct 30 03:30:35.994787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.994742 4560 generic.go:356] "Generic (PLEG): container finished" podID="6e43c67c-693c-4c85-84ed-0f38542ea858" containerID="ea6ea71a3a975d5bc8ab23ebae4ed87a21e6e7f6bfca0ae2876773a3b8f78d73" exitCode=0 Oct 30 03:30:35.994986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:35.994907 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" event={"ID":"6e43c67c-693c-4c85-84ed-0f38542ea858","Type":"ContainerDied","Data":"ea6ea71a3a975d5bc8ab23ebae4ed87a21e6e7f6bfca0ae2876773a3b8f78d73"} Oct 30 03:30:36.018149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.018121 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:30:36.143537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.143442 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\") pod \"6e43c67c-693c-4c85-84ed-0f38542ea858\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " Oct 30 03:30:36.143681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.143593 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\" (UniqueName: \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\") pod \"6e43c67c-693c-4c85-84ed-0f38542ea858\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " Oct 30 03:30:36.143681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.143659 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") pod \"6e43c67c-693c-4c85-84ed-0f38542ea858\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " Oct 30 03:30:36.143790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.143757 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\") pod \"6e43c67c-693c-4c85-84ed-0f38542ea858\" (UID: \"6e43c67c-693c-4c85-84ed-0f38542ea858\") " Oct 30 03:30:36.143939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.143854 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca" pod "6e43c67c-693c-4c85-84ed-0f38542ea858" (UID: "6e43c67c-693c-4c85-84ed-0f38542ea858"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:36.144205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.144182 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e43c67c-693c-4c85-84ed-0f38542ea858-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:36.146343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.146313 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs" pod "6e43c67c-693c-4c85-84ed-0f38542ea858" (UID: "6e43c67c-693c-4c85-84ed-0f38542ea858"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:36.146436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.146412 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z" pod "6e43c67c-693c-4c85-84ed-0f38542ea858" (UID: "6e43c67c-693c-4c85-84ed-0f38542ea858"). InnerVolumeSpecName "kube-api-access-lqw9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:36.146481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.146449 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls" pod "6e43c67c-693c-4c85-84ed-0f38542ea858" (UID: "6e43c67c-693c-4c85-84ed-0f38542ea858"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:36.245512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.245478 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw9z\" (UniqueName: \"kubernetes.io/projected/6e43c67c-693c-4c85-84ed-0f38542ea858-kube-api-access-lqw9z\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:36.245512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.245509 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:36.245512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:36.245529 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e43c67c-693c-4c85-84ed-0f38542ea858-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:37.001604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:37.001562 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" event={"ID":"6e43c67c-693c-4c85-84ed-0f38542ea858","Type":"ContainerDied","Data":"05da31b3a65315c955178a31da77416fb4a1b31721198234a193f23258eb1bd0"} Oct 30 03:30:37.001604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:37.001587 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6" Oct 30 03:30:37.002138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:37.001636 4560 scope.go:122] "RemoveContainer" containerID="ea6ea71a3a975d5bc8ab23ebae4ed87a21e6e7f6bfca0ae2876773a3b8f78d73" Oct 30 03:30:37.053356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:37.053318 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:30:37.065882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:37.065848 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-194506-5b89568b98-lhwt6"] Oct 30 03:30:38.152998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:38.152955 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e43c67c-693c-4c85-84ed-0f38542ea858" path="/var/lib/kubelet/pods/6e43c67c-693c-4c85-84ed-0f38542ea858/volumes" Oct 30 03:30:40.847622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.847595 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:30:40.998673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998639 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.998866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998683 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.998866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998742 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.998866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998786 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.998866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998811 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.999062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998910 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.999062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.998966 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:40.999062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999010 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.999062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999053 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.999275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999085 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\") pod \"b8031d3d-6736-42c6-8037-39550d24fae7\" (UID: \"b8031d3d-6736-42c6-8037-39550d24fae7\") " Oct 30 03:30:40.999275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999158 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:40.999275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999229 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:40.999455 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999387 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:40.999455 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999409 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:40.999719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999698 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:40.999802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999725 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:40.999802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999740 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:40.999802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999755 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8031d3d-6736-42c6-8037-39550d24fae7-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:40.999802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:40.999768 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:41.001226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.001200 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:41.001318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.001288 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:41.001431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.001408 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "kube-api-access-x8lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:41.001503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.001474 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo" pod "b8031d3d-6736-42c6-8037-39550d24fae7" (UID: "b8031d3d-6736-42c6-8037-39550d24fae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:30:41.031125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.031084 4560 generic.go:356] "Generic (PLEG): container finished" podID="b8031d3d-6736-42c6-8037-39550d24fae7" containerID="ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0" exitCode=0 Oct 30 03:30:41.031313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.031138 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" Oct 30 03:30:41.031313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.031178 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" event={"ID":"b8031d3d-6736-42c6-8037-39550d24fae7","Type":"ContainerDied","Data":"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0"} Oct 30 03:30:41.031313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.031246 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85" event={"ID":"b8031d3d-6736-42c6-8037-39550d24fae7","Type":"ContainerDied","Data":"fb77394f79b6a5e8ad562aa3c4e049c465e5c25997d296265cd3560462a8b9b6"} Oct 30 03:30:41.031313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.031277 4560 scope.go:122] "RemoveContainer" containerID="ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0" Oct 30 03:30:41.044008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.043984 4560 scope.go:122] "RemoveContainer" containerID="ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0" Oct 30 03:30:41.044376 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:41.044355 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0\": container with ID starting with ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0 not found: ID does not exist" containerID="ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0" Oct 30 03:30:41.044437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.044387 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0"} err="failed to get container status \"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0\": rpc error: code = NotFound desc = could not find container \"ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0\": container with ID starting with ac32bf13ad94a27d36cf6a4c6f55847f3e757e42505ce16212cda70869cfcbd0 not found: ID does not exist" Oct 30 03:30:41.084544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.084449 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:30:41.088086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.088063 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9lv85"] Oct 30 03:30:41.100740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.100707 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8031d3d-6736-42c6-8037-39550d24fae7-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:41.100740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.100736 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:41.100902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.100749 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8lhf\" (UniqueName: \"kubernetes.io/projected/b8031d3d-6736-42c6-8037-39550d24fae7-kube-api-access-x8lhf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:41.100902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:41.100761 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8031d3d-6736-42c6-8037-39550d24fae7-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:41.749208 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:41.749171 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:30:42.173599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:42.173207 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" path="/var/lib/kubelet/pods/b8031d3d-6736-42c6-8037-39550d24fae7/volumes" Oct 30 03:30:44.463988 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:44.463952 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:30:44.464406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:44.463958 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:30:45.723565 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.723534 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:30:45.751375 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.751336 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\" (UniqueName: \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\") pod \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " Oct 30 03:30:45.751573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.751461 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\") pod \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\" (UID: \"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6\") " Oct 30 03:30:45.752026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.751993 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume" pod "f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" (UID: "f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:45.753902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.753875 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb" pod "f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" (UID: "f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6"). InnerVolumeSpecName "kube-api-access-ftflb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:45.848911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.848885 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:30:45.852785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.852764 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftflb\" (UniqueName: \"kubernetes.io/projected/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-kube-api-access-ftflb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:45.852785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.852785 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:45.953803 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.953768 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\") pod \"99daeb88-5777-4162-a4c2-509a984e742e\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " Oct 30 03:30:45.953983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.953821 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\" (UniqueName: \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\") pod \"99daeb88-5777-4162-a4c2-509a984e742e\" (UID: \"99daeb88-5777-4162-a4c2-509a984e742e\") " Oct 30 03:30:45.954187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.954163 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume" pod "99daeb88-5777-4162-a4c2-509a984e742e" (UID: "99daeb88-5777-4162-a4c2-509a984e742e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:45.956078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:45.956049 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p" pod "99daeb88-5777-4162-a4c2-509a984e742e" (UID: "99daeb88-5777-4162-a4c2-509a984e742e"). InnerVolumeSpecName "kube-api-access-srv9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:46.055738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.055702 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99daeb88-5777-4162-a4c2-509a984e742e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:46.055738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.055730 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srv9p\" (UniqueName: \"kubernetes.io/projected/99daeb88-5777-4162-a4c2-509a984e742e-kube-api-access-srv9p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:30:46.061461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.061427 4560 generic.go:356] "Generic (PLEG): container finished" podID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerID="14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e" exitCode=0 Oct 30 03:30:46.061641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.061468 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" Oct 30 03:30:46.061641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.061512 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" event={"ID":"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6","Type":"ContainerDied","Data":"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e"} Oct 30 03:30:46.061641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.061581 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc" event={"ID":"f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6","Type":"ContainerDied","Data":"8cdc6648ad69eeb90d575a391750a5b2e8f5ddce8b97611149edde7f557efe88"} Oct 30 03:30:46.061641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.061603 4560 scope.go:122] "RemoveContainer" containerID="14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e" Oct 30 03:30:46.063306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.063278 4560 generic.go:356] "Generic (PLEG): container finished" podID="99daeb88-5777-4162-a4c2-509a984e742e" containerID="763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83" exitCode=0 Oct 30 03:30:46.063432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.063319 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" Oct 30 03:30:46.063432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.063358 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" event={"ID":"99daeb88-5777-4162-a4c2-509a984e742e","Type":"ContainerDied","Data":"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83"} Oct 30 03:30:46.063432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.063403 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd" event={"ID":"99daeb88-5777-4162-a4c2-509a984e742e","Type":"ContainerDied","Data":"e05a5fdf1c4faa02f81417a6f7834fa25cfb972accc9c0d0e82ffda29b4a8465"} Oct 30 03:30:46.071978 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.071960 4560 scope.go:122] "RemoveContainer" containerID="14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e" Oct 30 03:30:46.072271 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:46.072246 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e\": container with ID starting with 14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e not found: ID does not exist" containerID="14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e" Oct 30 03:30:46.072374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.072279 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e"} err="failed to get container status \"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e\": rpc error: code = NotFound desc = could not find container \"14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e\": container with ID starting with 14485994b3ac6323f59ba3518381c49af27a0221775b46c45f06cd2a33cc2a6e not found: ID does not exist" Oct 30 03:30:46.072374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.072297 4560 scope.go:122] "RemoveContainer" containerID="763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83" Oct 30 03:30:46.080755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.080737 4560 scope.go:122] "RemoveContainer" containerID="763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83" Oct 30 03:30:46.081020 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:30:46.081001 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83\": container with ID starting with 763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83 not found: ID does not exist" containerID="763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83" Oct 30 03:30:46.081080 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.081035 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83"} err="failed to get container status \"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83\": rpc error: code = NotFound desc = could not find container \"763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83\": container with ID starting with 763df0371d709681cb5c9584fe78b056674142d52222ee1e299ad47b6fdb9c83 not found: ID does not exist" Oct 30 03:30:46.114013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.113939 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:30:46.119158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.119136 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-194506-6796d89f95-ckkhd"] Oct 30 03:30:46.154955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.154920 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99daeb88-5777-4162-a4c2-509a984e742e" path="/var/lib/kubelet/pods/99daeb88-5777-4162-a4c2-509a984e742e/volumes" Oct 30 03:30:46.155240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.155225 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:30:46.156914 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:46.156889 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-194506-5d7c4bb67d-br4cc"] Oct 30 03:30:48.151653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:30:48.151625 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" path="/var/lib/kubelet/pods/f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6/volumes" Oct 30 03:31:04.446300 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.446266 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:31:04.446797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.446475 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" podUID="5527d7f6-4880-4bb5-961d-584f8af08d5a" containerName="maas-api" containerID="cri-o://d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c" gracePeriod=30 Oct 30 03:31:04.731379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.731347 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:31:04.868408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.868377 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\") pod \"5527d7f6-4880-4bb5-961d-584f8af08d5a\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " Oct 30 03:31:04.868590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.868431 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\") pod \"5527d7f6-4880-4bb5-961d-584f8af08d5a\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " Oct 30 03:31:04.868590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.868475 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\") pod \"5527d7f6-4880-4bb5-961d-584f8af08d5a\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " Oct 30 03:31:04.868590 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.868566 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\" (UniqueName: \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\") pod \"5527d7f6-4880-4bb5-961d-584f8af08d5a\" (UID: \"5527d7f6-4880-4bb5-961d-584f8af08d5a\") " Oct 30 03:31:04.868900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.868864 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca" pod "5527d7f6-4880-4bb5-961d-584f8af08d5a" (UID: "5527d7f6-4880-4bb5-961d-584f8af08d5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:04.870510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.870480 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs" pod "5527d7f6-4880-4bb5-961d-584f8af08d5a" (UID: "5527d7f6-4880-4bb5-961d-584f8af08d5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:31:04.870669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.870646 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls" pod "5527d7f6-4880-4bb5-961d-584f8af08d5a" (UID: "5527d7f6-4880-4bb5-961d-584f8af08d5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:31:04.870879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.870859 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n" pod "5527d7f6-4880-4bb5-961d-584f8af08d5a" (UID: "5527d7f6-4880-4bb5-961d-584f8af08d5a"). InnerVolumeSpecName "kube-api-access-spv2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:04.969717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.969688 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.969717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.969716 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527d7f6-4880-4bb5-961d-584f8af08d5a-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.969717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.969725 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5527d7f6-4880-4bb5-961d-584f8af08d5a-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.969953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:04.969734 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spv2n\" (UniqueName: \"kubernetes.io/projected/5527d7f6-4880-4bb5-961d-584f8af08d5a-kube-api-access-spv2n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:05.178355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.178261 4560 generic.go:356] "Generic (PLEG): container finished" podID="5527d7f6-4880-4bb5-961d-584f8af08d5a" containerID="d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c" exitCode=0 Oct 30 03:31:05.178355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.178318 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" Oct 30 03:31:05.178634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.178350 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" event={"ID":"5527d7f6-4880-4bb5-961d-584f8af08d5a","Type":"ContainerDied","Data":"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c"} Oct 30 03:31:05.178634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.178403 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q" event={"ID":"5527d7f6-4880-4bb5-961d-584f8af08d5a","Type":"ContainerDied","Data":"dac82767cc2b637787273ec0051b4017ac6a2a7841db639448f6673355145c35"} Oct 30 03:31:05.178634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.178425 4560 scope.go:122] "RemoveContainer" containerID="d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c" Oct 30 03:31:05.188509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.188493 4560 scope.go:122] "RemoveContainer" containerID="d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c" Oct 30 03:31:05.188796 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:31:05.188778 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c\": container with ID starting with d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c not found: ID does not exist" containerID="d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c" Oct 30 03:31:05.188863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.188804 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c"} err="failed to get container status \"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c\": rpc error: code = NotFound desc = could not find container \"d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c\": container with ID starting with d2130a75c0fbf094d00c72ae30cb1d4cc7bcab3458fc522e1b798ef5c3b91a9c not found: ID does not exist" Oct 30 03:31:05.230397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.230371 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:31:05.243113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.243075 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-773fbb-696bbbc6b5-l547q"] Oct 30 03:31:05.284047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284010 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6"] Oct 30 03:31:05.284596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284576 4560 cpu_manager.go:414] "containerMap: removing container" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.284596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284596 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284612 4560 cpu_manager.go:414] "containerMap: removing container" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284619 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284629 4560 cpu_manager.go:414] "containerMap: removing container" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284638 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284648 4560 cpu_manager.go:414] "containerMap: removing container" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284655 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284665 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284673 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284691 4560 cpu_manager.go:414] "containerMap: removing container" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284699 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284712 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284719 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284735 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.284741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284741 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284767 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284775 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284808 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5527d7f6-4880-4bb5-961d-584f8af08d5a" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284817 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5527d7f6-4880-4bb5-961d-584f8af08d5a" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284833 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284841 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284863 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6e43c67c-693c-4c85-84ed-0f38542ea858" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284871 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e43c67c-693c-4c85-84ed-0f38542ea858" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284888 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.284896 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285017 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b91c87-dd7c-4a34-8e5c-e5fabb77afc6" containerName="payload-processing" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285030 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285040 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285047 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285055 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285065 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285072 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8031d3d-6736-42c6-8037-39550d24fae7" containerName="istio-proxy" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285080 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285088 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5527d7f6-4880-4bb5-961d-584f8af08d5a" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285098 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="99daeb88-5777-4162-a4c2-509a984e742e" containerName="payload-pre-processing" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285106 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285113 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285122 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285132 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e43c67c-693c-4c85-84ed-0f38542ea858" containerName="maas-api" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285140 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:05.285454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.285148 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:05.290212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.290189 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.308288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.308260 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:31:05.308454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.308314 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:31:05.308454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.308328 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:31:05.308454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.308374 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:31:05.312565 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.311744 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6"] Oct 30 03:31:05.373493 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.373456 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.373699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.373678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9w5\" (UniqueName: \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.373768 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.373731 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.474967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.474923 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9w5\" (UniqueName: \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.475437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.475036 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.475437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.475098 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.475743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.475723 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.477468 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.477448 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.494577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.494555 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9w5\" (UniqueName: \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\") pod \"maas-api-revoke-sub-3ed1e115-r8ps6\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.604762 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.604730 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:05.904494 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:05.904412 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6"] Oct 30 03:31:05.909583 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:31:05.909554 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b27d28_33b5_48bd_9387_4f9be796c000.slice/crio-7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0 WatchSource:0}: Error finding container 7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0: Status 404 returned error can't find the container with id 7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0 Oct 30 03:31:06.152200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:06.152168 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5527d7f6-4880-4bb5-961d-584f8af08d5a" path="/var/lib/kubelet/pods/5527d7f6-4880-4bb5-961d-584f8af08d5a/volumes" Oct 30 03:31:06.185197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:06.185160 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" event={"ID":"81b27d28-33b5-48bd-9387-4f9be796c000","Type":"ContainerStarted","Data":"d9ee0f22a1cbefaaec5d5d6898d0b5ccbdedd5de7b45645f6153d1a2ccbb1da4"} Oct 30 03:31:06.185385 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:06.185224 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" event={"ID":"81b27d28-33b5-48bd-9387-4f9be796c000","Type":"ContainerStarted","Data":"7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0"} Oct 30 03:31:06.226092 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:06.225495 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" podStartSLOduration=1.225479282 podStartE2EDuration="1.225479282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:06.224903828 +0000 UTC m=+1694.210023718" watchObservedRunningTime="2026-10-30 03:31:06.225479282 +0000 UTC m=+1694.210599172" Oct 30 03:31:07.194984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:07.194945 4560 generic.go:356] "Generic (PLEG): container finished" podID="81b27d28-33b5-48bd-9387-4f9be796c000" containerID="d9ee0f22a1cbefaaec5d5d6898d0b5ccbdedd5de7b45645f6153d1a2ccbb1da4" exitCode=0 Oct 30 03:31:07.195662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:07.195040 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" event={"ID":"81b27d28-33b5-48bd-9387-4f9be796c000","Type":"ContainerDied","Data":"d9ee0f22a1cbefaaec5d5d6898d0b5ccbdedd5de7b45645f6153d1a2ccbb1da4"} Oct 30 03:31:08.345604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.345576 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:08.407775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.407738 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\") pod \"81b27d28-33b5-48bd-9387-4f9be796c000\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " Oct 30 03:31:08.407939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.407823 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\") pod \"81b27d28-33b5-48bd-9387-4f9be796c000\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " Oct 30 03:31:08.407986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.407950 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\" (UniqueName: \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\") pod \"81b27d28-33b5-48bd-9387-4f9be796c000\" (UID: \"81b27d28-33b5-48bd-9387-4f9be796c000\") " Oct 30 03:31:08.408162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.408133 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca" pod "81b27d28-33b5-48bd-9387-4f9be796c000" (UID: "81b27d28-33b5-48bd-9387-4f9be796c000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:08.408544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.408498 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:08.410038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.410010 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca" pod "81b27d28-33b5-48bd-9387-4f9be796c000" (UID: "81b27d28-33b5-48bd-9387-4f9be796c000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:31:08.410453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.410433 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5" pod "81b27d28-33b5-48bd-9387-4f9be796c000" (UID: "81b27d28-33b5-48bd-9387-4f9be796c000"). InnerVolumeSpecName "kube-api-access-nn9w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:08.509195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.509106 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b27d28-33b5-48bd-9387-4f9be796c000-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:08.509195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:08.509139 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9w5\" (UniqueName: \"kubernetes.io/projected/81b27d28-33b5-48bd-9387-4f9be796c000-kube-api-access-nn9w5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:09.207453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:09.207410 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" event={"ID":"81b27d28-33b5-48bd-9387-4f9be796c000","Type":"ContainerDied","Data":"7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0"} Oct 30 03:31:09.207453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:09.207451 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6" Oct 30 03:31:09.207690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:09.207468 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d95e54a50f2e0da436b7f7dfa9569301bcd955aa6c9151a4e0a1d84849503b0" Oct 30 03:31:11.750660 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:31:11.750615 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:31:20.558800 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.558761 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj"] Oct 30 03:31:20.559274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559259 4560 cpu_manager.go:414] "containerMap: removing container" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:20.559316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559274 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947cb936-ce35-431d-ac15-9116c736fda5" containerName="cleanup" Oct 30 03:31:20.559316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559283 4560 cpu_manager.go:414] "containerMap: removing container" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:20.559316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559289 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769aea49-1206-430d-9963-a780b46b2ae9" containerName="cleanup" Oct 30 03:31:20.559316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559303 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:20.559316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559312 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559326 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559332 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac0cd33-54bb-44e6-b25d-1d3ce784669e" containerName="cleanup" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559341 4560 cpu_manager.go:414] "containerMap: removing container" podUID="81b27d28-33b5-48bd-9387-4f9be796c000" containerName="revoke-keys" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559346 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b27d28-33b5-48bd-9387-4f9be796c000" containerName="revoke-keys" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559450 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c445d-7d72-4e1f-9289-760d7353a9b1" containerName="cleanup" Oct 30 03:31:20.559477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.559462 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b27d28-33b5-48bd-9387-4f9be796c000" containerName="revoke-keys" Oct 30 03:31:20.561672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.561656 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.576793 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.576762 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:31:20.589712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.589618 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:31:20.589712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.589684 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:31:20.589712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.589688 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:31:20.592443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.592420 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj"] Oct 30 03:31:20.641924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.641893 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ml4\" (UniqueName: \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.642119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.642023 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.642119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.642067 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.742811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.742774 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ml4\" (UniqueName: \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.743000 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.742868 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.743000 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.742890 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.743693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.743661 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.745212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.745192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.762465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.762444 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ml4\" (UniqueName: \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\") pod \"maas-api-revoke-sub-c990a35a-2m6wj\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:20.873031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:20.872947 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:21.175698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:21.175659 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj"] Oct 30 03:31:21.176833 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:31:21.176804 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f4bf0c_3c6f_47a1_84ac_766125ed652a.slice/crio-849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b WatchSource:0}: Error finding container 849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b: Status 404 returned error can't find the container with id 849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b Oct 30 03:31:21.271897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:21.271863 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" event={"ID":"19f4bf0c-3c6f-47a1-84ac-766125ed652a","Type":"ContainerStarted","Data":"849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b"} Oct 30 03:31:22.278383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:22.278348 4560 generic.go:356] "Generic (PLEG): container finished" podID="19f4bf0c-3c6f-47a1-84ac-766125ed652a" containerID="b7423e993f07e15473a8faeec8a803632add28a4fbf45febf80dcf27937aa260" exitCode=0 Oct 30 03:31:22.278800 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:22.278421 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" event={"ID":"19f4bf0c-3c6f-47a1-84ac-766125ed652a","Type":"ContainerDied","Data":"b7423e993f07e15473a8faeec8a803632add28a4fbf45febf80dcf27937aa260"} Oct 30 03:31:23.426109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.426084 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:23.468711 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.468678 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\") pod \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " Oct 30 03:31:23.468870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.468754 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\" (UniqueName: \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\") pod \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " Oct 30 03:31:23.468870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.468807 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\") pod \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\" (UID: \"19f4bf0c-3c6f-47a1-84ac-766125ed652a\") " Oct 30 03:31:23.469107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.469083 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca" pod "19f4bf0c-3c6f-47a1-84ac-766125ed652a" (UID: "19f4bf0c-3c6f-47a1-84ac-766125ed652a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:23.469200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.469185 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:23.470902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.470876 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4" pod "19f4bf0c-3c6f-47a1-84ac-766125ed652a" (UID: "19f4bf0c-3c6f-47a1-84ac-766125ed652a"). InnerVolumeSpecName "kube-api-access-q5ml4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:23.471003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.470926 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca" pod "19f4bf0c-3c6f-47a1-84ac-766125ed652a" (UID: "19f4bf0c-3c6f-47a1-84ac-766125ed652a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:31:23.569997 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.569907 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ml4\" (UniqueName: \"kubernetes.io/projected/19f4bf0c-3c6f-47a1-84ac-766125ed652a-kube-api-access-q5ml4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:23.569997 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:23.569941 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f4bf0c-3c6f-47a1-84ac-766125ed652a-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:31:24.292532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:24.292497 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" Oct 30 03:31:24.292706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:24.292497 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj" event={"ID":"19f4bf0c-3c6f-47a1-84ac-766125ed652a","Type":"ContainerDied","Data":"849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b"} Oct 30 03:31:24.292706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:31:24.292636 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="849b7c4f2bda89e47a5c02b4ec80910cf89c2da54fbf01cdd3765111aeead87b" Oct 30 03:31:41.749288 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:31:41.749251 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:32:11.749774 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:32:11.749738 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:32:41.749207 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:32:41.749168 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:32:45.872459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:45.872419 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:32:45.872892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:45.872764 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" containerID="cri-o://26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639" gracePeriod=30 Oct 30 03:32:51.138561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.138473 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:32:51.235490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235457 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235549 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235573 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235598 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235627 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235655 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235762 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235803 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235871 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:51.235972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235911 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\") pod \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\" (UID: \"d4f799c8-bc83-436d-b06d-01d6bc8695c6\") " Oct 30 03:32:51.235972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235953 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:51.236236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.235988 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:32:51.236236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236109 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:51.236236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236167 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:51.236642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236570 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.236642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236596 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.236642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236612 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.236642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236625 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.236642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.236637 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.238365 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.238334 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "kube-api-access-72hpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:32:51.238591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.238560 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:32:51.238694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.238623 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:51.238775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.238760 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo" pod "d4f799c8-bc83-436d-b06d-01d6bc8695c6" (UID: "d4f799c8-bc83-436d-b06d-01d6bc8695c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:32:51.337866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.337826 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.337866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.337854 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hpc\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-kube-api-access-72hpc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.337866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.337863 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.337866 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.337872 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f799c8-bc83-436d-b06d-01d6bc8695c6-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:32:51.762862 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.762823 4560 generic.go:356] "Generic (PLEG): container finished" podID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerID="26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639" exitCode=0 Oct 30 03:32:51.763049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.762892 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" Oct 30 03:32:51.763049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.762907 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" event={"ID":"d4f799c8-bc83-436d-b06d-01d6bc8695c6","Type":"ContainerDied","Data":"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639"} Oct 30 03:32:51.763049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.762969 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq" event={"ID":"d4f799c8-bc83-436d-b06d-01d6bc8695c6","Type":"ContainerDied","Data":"997b981536fcf55cb39c56f777633fda25ef2495ed0cc80b751906b1c2019f7e"} Oct 30 03:32:51.763049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.762991 4560 scope.go:122] "RemoveContainer" containerID="26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639" Oct 30 03:32:51.772979 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.772955 4560 scope.go:122] "RemoveContainer" containerID="26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639" Oct 30 03:32:51.773304 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:32:51.773285 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639\": container with ID starting with 26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639 not found: ID does not exist" containerID="26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639" Oct 30 03:32:51.773353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.773316 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639"} err="failed to get container status \"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639\": rpc error: code = NotFound desc = could not find container \"26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639\": container with ID starting with 26300175db60f70e45f0fc5c67c807279abcf7cb18d9f31dc76a0f8f48cfc639 not found: ID does not exist" Oct 30 03:32:51.829046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.829004 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:32:51.842380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:51.842326 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-zgbkq"] Oct 30 03:32:52.153097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:32:52.153015 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" path="/var/lib/kubelet/pods/d4f799c8-bc83-436d-b06d-01d6bc8695c6/volumes" Oct 30 03:33:11.749176 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:33:11.749117 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:33:16.029373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:16.029332 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z"] Oct 30 03:33:16.040426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:16.040394 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b745e537-6bs9z"] Oct 30 03:33:16.152445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:16.152414 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5666d11b-373f-4404-9609-de88f1a72105" path="/var/lib/kubelet/pods/5666d11b-373f-4404-9609-de88f1a72105/volumes" Oct 30 03:33:20.997167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:20.997120 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:33:20.997669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:20.997364 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" containerID="cri-o://93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22" gracePeriod=30 Oct 30 03:33:26.043153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.043122 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6"] Oct 30 03:33:26.055373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.055345 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369cb975-7rkd6"] Oct 30 03:33:26.153027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.152989 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a48880-35b8-4793-9472-18fb9584693f" path="/var/lib/kubelet/pods/39a48880-35b8-4793-9472-18fb9584693f/volumes" Oct 30 03:33:26.258861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.258836 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:33:26.309696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.309599 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.309696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.309650 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.309696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.309695 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310059 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:33:26.310158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310114 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310196 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310236 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310314 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310391 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310344 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.310493 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310470 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:26.310588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.310493 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\") pod \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\" (UID: \"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a\") " Oct 30 03:33:26.311056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311029 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:26.311201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311083 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:26.311282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311109 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:26.311404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311391 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.311597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311580 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.311597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311599 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.311755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311612 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.311755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.311627 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.313390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.313345 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:33:26.313718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.313695 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "kube-api-access-8r9lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:33:26.314025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.314007 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:33:26.314253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.314226 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy" pod "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" (UID: "6a2ef183-6a0f-4429-bd85-0c4d5f0d807a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:26.412751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.412717 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.412751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.412751 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9lb\" (UniqueName: \"kubernetes.io/projected/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-kube-api-access-8r9lb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.412970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.412764 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.412970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.412778 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:33:26.951950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.951919 4560 generic.go:356] "Generic (PLEG): container finished" podID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerID="93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22" exitCode=0 Oct 30 03:33:26.952138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.951970 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" Oct 30 03:33:26.952138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.952010 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" event={"ID":"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a","Type":"ContainerDied","Data":"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22"} Oct 30 03:33:26.952138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.952070 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx" event={"ID":"6a2ef183-6a0f-4429-bd85-0c4d5f0d807a","Type":"ContainerDied","Data":"c53ae1738eea43e86128c7a5cbc0879fc61f99ce15d7f7b095346b9984e927bf"} Oct 30 03:33:26.952138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.952101 4560 scope.go:122] "RemoveContainer" containerID="93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22" Oct 30 03:33:26.962082 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.962062 4560 scope.go:122] "RemoveContainer" containerID="93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22" Oct 30 03:33:26.962353 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:33:26.962336 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22\": container with ID starting with 93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22 not found: ID does not exist" containerID="93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22" Oct 30 03:33:26.962401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:26.962363 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22"} err="failed to get container status \"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22\": rpc error: code = NotFound desc = could not find container \"93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22\": container with ID starting with 93d0642cc41d8c381d39d549ee0073b7a46ea07dc5fc77ee8be287264faa0a22 not found: ID does not exist" Oct 30 03:33:27.019533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:27.019492 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:33:27.032264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:27.032233 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gkhfx"] Oct 30 03:33:28.151201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:33:28.151170 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" path="/var/lib/kubelet/pods/6a2ef183-6a0f-4429-bd85-0c4d5f0d807a/volumes" Oct 30 03:33:41.749460 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:33:41.749371 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:34:11.749573 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:34:11.749536 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:34:41.750386 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:34:41.750351 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:34:52.735825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:34:52.735797 4560 scope.go:122] "RemoveContainer" containerID="780263b443463821c78b95bde9f6758b92ace327d07ac1c90e41bd2e6d823abc" Oct 30 03:34:52.744687 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:34:52.744661 4560 scope.go:122] "RemoveContainer" containerID="a69a5a6dfdf80fb3571edbabe533263c71b646fa7a87341ba2ae9af84ed86638" Oct 30 03:35:10.593226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593142 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 03:35:10.593708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593683 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" Oct 30 03:35:10.593708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593698 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" Oct 30 03:35:10.593708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593710 4560 cpu_manager.go:414] "containerMap: removing container" podUID="19f4bf0c-3c6f-47a1-84ac-766125ed652a" containerName="revoke-keys" Oct 30 03:35:10.593817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593715 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f4bf0c-3c6f-47a1-84ac-766125ed652a" containerName="revoke-keys" Oct 30 03:35:10.593817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593732 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" Oct 30 03:35:10.593817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593738 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" Oct 30 03:35:10.593817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593816 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f4bf0c-3c6f-47a1-84ac-766125ed652a" containerName="revoke-keys" Oct 30 03:35:10.593939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593829 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2ef183-6a0f-4429-bd85-0c4d5f0d807a" containerName="istio-proxy" Oct 30 03:35:10.593939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.593838 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f799c8-bc83-436d-b06d-01d6bc8695c6" containerName="istio-proxy" Oct 30 03:35:10.598616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.598595 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.626941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.626910 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8gxpp\"" Oct 30 03:35:10.627139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.626919 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7d99a4-serving-cert\"" Oct 30 03:35:10.627220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.626924 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:35:10.627220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.626975 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:35:10.627220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.627157 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 03:35:10.631089 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.631055 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 03:35:10.662899 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.662864 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.663093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.662992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.663093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.663080 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.663221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.663126 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhngn\" (UniqueName: \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.764528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.764488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.764715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.764610 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.764715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.764653 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.764715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.764676 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhngn\" (UniqueName: \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.765401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.765372 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.767104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.767081 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.767211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.767135 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.784698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.784670 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhngn\" (UniqueName: \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\") pod \"maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:10.912364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:10.912267 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:11.221734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:11.221696 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 03:35:11.224924 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:35:11.224895 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f8c88a_6b87_4e53_8b33_ed348f369c7b.slice/crio-a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7 WatchSource:0}: Error finding container a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7: Status 404 returned error can't find the container with id a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7 Oct 30 03:35:11.226283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:11.226256 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:35:11.517504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:11.517420 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" event={"ID":"f3f8c88a-6b87-4e53-8b33-ed348f369c7b","Type":"ContainerStarted","Data":"a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7"} Oct 30 03:35:11.763642 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:35:11.763598 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:35:12.524228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:12.524186 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" event={"ID":"f3f8c88a-6b87-4e53-8b33-ed348f369c7b","Type":"ContainerStarted","Data":"5da0dd9efab6d9517fa816e4a7517355bf28fa8e7ddd666905e539db351aeebc"} Oct 30 03:35:12.524409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:12.524348 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:12.567332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:12.567280 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" podStartSLOduration=2.193728381 podStartE2EDuration="2.567261912s" totalImagesPullingTime="373.533531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:12.564902151 +0000 UTC m=+1940.550022052" watchObservedRunningTime="2026-10-30 03:35:12.567261912 +0000 UTC m=+1940.552381801" Oct 30 03:35:18.535657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:18.535623 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 03:35:20.378286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.378250 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 03:35:20.382374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.382350 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.409651 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.409602 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-6ad168-serving-cert\"" Oct 30 03:35:20.415275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.415243 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 03:35:20.489171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.489112 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.489356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.489203 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.489356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.489267 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.489482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.489457 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj9v\" (UniqueName: \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.590423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.590306 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.590423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.590395 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj9v\" (UniqueName: \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:35:20.590462 4560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-serving-cert: secret "maas-api-e2e-shared-a-w0-6ad168-serving-cert" not found Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.590498 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:35:20.590568 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls podName:28f93420-c8bc-44f8-a5f3-d0d4ad165920 nodeName:}" failed. No retries permitted until 2026-10-30 03:35:21.090545264 +0000 UTC m=+1949.075665147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls") pod "maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" (UID: "28f93420-c8bc-44f8-a5f3-d0d4ad165920") : secret "maas-api-e2e-shared-a-w0-6ad168-serving-cert" not found Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.590628 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.591266 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.595635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.592984 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:20.610298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:20.610276 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj9v\" (UniqueName: \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:21.095808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:21.095764 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:21.098409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:21.098375 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:21.297977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:21.297931 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:21.631758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:21.631729 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 03:35:21.633240 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:35:21.633210 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f93420_c8bc_44f8_a5f3_d0d4ad165920.slice/crio-e86601f75bd30cc950cae2606be7d70b19e8b276e83da3a9fc253849ca8dc224 WatchSource:0}: Error finding container e86601f75bd30cc950cae2606be7d70b19e8b276e83da3a9fc253849ca8dc224: Status 404 returned error can't find the container with id e86601f75bd30cc950cae2606be7d70b19e8b276e83da3a9fc253849ca8dc224 Oct 30 03:35:22.584770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:22.584737 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" event={"ID":"28f93420-c8bc-44f8-a5f3-d0d4ad165920","Type":"ContainerStarted","Data":"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c"} Oct 30 03:35:22.584975 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:22.584790 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" event={"ID":"28f93420-c8bc-44f8-a5f3-d0d4ad165920","Type":"ContainerStarted","Data":"e86601f75bd30cc950cae2606be7d70b19e8b276e83da3a9fc253849ca8dc224"} Oct 30 03:35:22.584975 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:22.584829 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:22.626507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:22.626451 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" podStartSLOduration=2.364724069 podStartE2EDuration="2.626436733s" totalImagesPullingTime="261.712664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:22.624270524 +0000 UTC m=+1950.609390409" watchObservedRunningTime="2026-10-30 03:35:22.626436733 +0000 UTC m=+1950.611556620" Oct 30 03:35:28.091431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.091392 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 03:35:28.095240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.095223 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.126408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.126375 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-547a78-serving-cert\"" Oct 30 03:35:28.129093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.129065 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 03:35:28.169604 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.169569 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.169779 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.169690 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98897\" (UniqueName: \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.169832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.169800 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.169832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.169826 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.271037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.270999 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.271210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.271047 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.271210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.271105 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.271210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.271168 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98897\" (UniqueName: \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.271725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.271698 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.273684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.273659 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.273802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.273712 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.292810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.292784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98897\" (UniqueName: \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\") pod \"maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.408892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.408803 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:28.596019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.595990 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 03:35:28.710290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:28.710258 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 03:35:28.715064 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:35:28.715027 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db61479_cb23_4486_9b05_f99634f64df7.slice/crio-c00f73fd8840770d0da11dcb5e545d3464edab0a966a8f510a194195d1750fb5 WatchSource:0}: Error finding container c00f73fd8840770d0da11dcb5e545d3464edab0a966a8f510a194195d1750fb5: Status 404 returned error can't find the container with id c00f73fd8840770d0da11dcb5e545d3464edab0a966a8f510a194195d1750fb5 Oct 30 03:35:29.623405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:29.623366 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" event={"ID":"7db61479-cb23-4486-9b05-f99634f64df7","Type":"ContainerStarted","Data":"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2"} Oct 30 03:35:29.623846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:29.623436 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" event={"ID":"7db61479-cb23-4486-9b05-f99634f64df7","Type":"ContainerStarted","Data":"c00f73fd8840770d0da11dcb5e545d3464edab0a966a8f510a194195d1750fb5"} Oct 30 03:35:29.623846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:29.623489 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:29.666021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:29.665965 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" podStartSLOduration=1.282882437 podStartE2EDuration="1.665946636s" totalImagesPullingTime="383.064199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:29.663757403 +0000 UTC m=+1957.648877290" watchObservedRunningTime="2026-10-30 03:35:29.665946636 +0000 UTC m=+1957.651066524" Oct 30 03:35:35.634696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:35.634663 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 03:35:41.748973 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:35:41.748934 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:35:54.574727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.574689 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:35:54.580093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.580063 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.609581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.609531 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"openshift-service-ca.crt\"" Oct 30 03:35:54.609742 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.609701 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"default-dockercfg-mwqzk\"" Oct 30 03:35:54.609798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.609763 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"kube-root-ca.crt\"" Oct 30 03:35:54.609908 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.609886 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"auth-test-model-da4553-kserve-self-signed-certs\"" Oct 30 03:35:54.613441 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.613414 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:35:54.763272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763239 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.763492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763280 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.763492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763315 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.763492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763409 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.763729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763558 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.763729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.763611 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rq7\" (UniqueName: \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.864946 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865017 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rq7\" (UniqueName: \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865107 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865154 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865186 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865615 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.865684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.865665 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.867442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.867414 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.868048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.868026 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.884355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.884333 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rq7\" (UniqueName: \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\") pod \"auth-test-model-da4553-kserve-64f5cc7d5-wxfz5\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:54.896267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:54.896241 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:55.199167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:55.199146 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:35:55.200789 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:35:55.200762 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5c9c5c3_f5ae_479b_9180_44c8c238812b.slice/crio-b283171a1d37a930abd6f09a87fc0aa309e9861b3447833286241afec238ed3b WatchSource:0}: Error finding container b283171a1d37a930abd6f09a87fc0aa309e9861b3447833286241afec238ed3b: Status 404 returned error can't find the container with id b283171a1d37a930abd6f09a87fc0aa309e9861b3447833286241afec238ed3b Oct 30 03:35:55.771959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:55.771919 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" event={"ID":"d5c9c5c3-f5ae-479b-9180-44c8c238812b","Type":"ContainerStarted","Data":"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799"} Oct 30 03:35:55.771959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:55.771960 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" event={"ID":"d5c9c5c3-f5ae-479b-9180-44c8c238812b","Type":"ContainerStarted","Data":"b283171a1d37a930abd6f09a87fc0aa309e9861b3447833286241afec238ed3b"} Oct 30 03:35:55.813285 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:55.813231 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" podStartSLOduration=1.8132137259999999 podStartE2EDuration="1.813213726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:55.811591601 +0000 UTC m=+1983.796711488" watchObservedRunningTime="2026-10-30 03:35:55.813213726 +0000 UTC m=+1983.798333615" Oct 30 03:35:55.896960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:55.896934 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:35:57.430883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.430850 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:35:57.434824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.434801 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.464995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.464966 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"openshift-service-ca.crt\"" Oct 30 03:35:57.465197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.465012 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"auth-test-model-1a1b45-kserve-self-signed-certs\"" Oct 30 03:35:57.465197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.465122 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"default-dockercfg-z9vxb\"" Oct 30 03:35:57.465327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.465217 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"kube-root-ca.crt\"" Oct 30 03:35:57.467607 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.467581 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:35:57.493314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493280 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.493453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493348 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.493453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493380 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.493598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493553 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfb6x\" (UniqueName: \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.493657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493610 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.493728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.493712 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.594760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.594728 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.594968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.594813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.594968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.594893 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.594968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.594926 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.594968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.594957 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.595180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.595054 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfb6x\" (UniqueName: \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.595354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.595330 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.595471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.595370 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.595471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.595455 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.597426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.597393 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.597810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.597790 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.614329 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.614285 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfb6x\" (UniqueName: \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\") pod \"auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:57.748113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:57.748078 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:35:58.054560 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:58.054529 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:35:58.055772 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:35:58.055742 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2e1874d_4482_42d2_b054_93e7ebdace49.slice/crio-8f7772ec9993b35bad823c2c15270ec1db6962dda44d33763f81d044615230e0 WatchSource:0}: Error finding container 8f7772ec9993b35bad823c2c15270ec1db6962dda44d33763f81d044615230e0: Status 404 returned error can't find the container with id 8f7772ec9993b35bad823c2c15270ec1db6962dda44d33763f81d044615230e0 Oct 30 03:35:58.793778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:58.793739 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" event={"ID":"e2e1874d-4482-42d2-b054-93e7ebdace49","Type":"ContainerStarted","Data":"69f2a6151a6e2790a7b1faf2d2cb044902ce1107865cdbd30a1fc8261b30c86d"} Oct 30 03:35:58.794168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:58.793788 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" event={"ID":"e2e1874d-4482-42d2-b054-93e7ebdace49","Type":"ContainerStarted","Data":"8f7772ec9993b35bad823c2c15270ec1db6962dda44d33763f81d044615230e0"} Oct 30 03:35:58.832919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:58.832867 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" podStartSLOduration=1.8328541550000002 podStartE2EDuration="1.832854155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:58.831955738 +0000 UTC m=+1986.817075630" watchObservedRunningTime="2026-10-30 03:35:58.832854155 +0000 UTC m=+1986.817974043" Oct 30 03:35:59.748302 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:35:59.748254 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:36:04.897281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:04.897245 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:36:04.910305 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:04.910268 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:36:05.546842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.546801 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 03:36:05.550703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.550681 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.571706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.571673 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 03:36:05.576735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.576704 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-817531-serving-cert\"" Oct 30 03:36:05.585884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.585850 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.586040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.585909 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.586132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.586105 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.586264 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.586244 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7558\" (UniqueName: \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.688013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.687976 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.688214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.688058 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7558\" (UniqueName: \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.688214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.688122 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.688214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.688149 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.688401 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:05.688280 4560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-serving-cert: secret "maas-api-e2e-shared-b-w0-817531-serving-cert" not found Oct 30 03:36:05.688401 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:05.688349 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls podName:e96b92c1-f79a-46e3-8ecf-74323753dd00 nodeName:}" failed. No retries permitted until 2026-10-30 03:36:06.188328695 +0000 UTC m=+1994.173448566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls") pod "maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" (UID: "e96b92c1-f79a-46e3-8ecf-74323753dd00") : secret "maas-api-e2e-shared-b-w0-817531-serving-cert" not found Oct 30 03:36:05.688788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.688764 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.690539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.690506 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.735653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.735542 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7558\" (UniqueName: \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:05.847152 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:05.847073 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:36:06.193477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:06.193379 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:06.195955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:06.195926 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:06.465176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:06.465141 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:06.783327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:06.783218 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 03:36:06.786029 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:36:06.785997 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96b92c1_f79a_46e3_8ecf_74323753dd00.slice/crio-906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842 WatchSource:0}: Error finding container 906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842: Status 404 returned error can't find the container with id 906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842 Oct 30 03:36:06.842549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:06.842496 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" event={"ID":"e96b92c1-f79a-46e3-8ecf-74323753dd00","Type":"ContainerStarted","Data":"906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842"} Oct 30 03:36:07.748885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.748845 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:36:07.762125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.762098 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:36:07.850437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.850397 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" event={"ID":"e96b92c1-f79a-46e3-8ecf-74323753dd00","Type":"ContainerStarted","Data":"2563b0e013faf340af012ef41b5168f3c64de11542b52c050c46544b164113ef"} Oct 30 03:36:07.850637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.850608 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:07.861853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.861820 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:36:07.895634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:07.895583 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" podStartSLOduration=2.5809673699999998 podStartE2EDuration="2.895563933s" totalImagesPullingTime="314.596563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:07.892905506 +0000 UTC m=+1995.878025394" watchObservedRunningTime="2026-10-30 03:36:07.895563933 +0000 UTC m=+1995.880683825" Oct 30 03:36:09.028486 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:09.028449 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6"] Oct 30 03:36:09.041005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:09.040972 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed1e115-r8ps6"] Oct 30 03:36:10.154142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:10.154100 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b27d28-33b5-48bd-9387-4f9be796c000" path="/var/lib/kubelet/pods/81b27d28-33b5-48bd-9387-4f9be796c000/volumes" Oct 30 03:36:11.756329 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:11.756281 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:36:13.864426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:13.864396 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 03:36:14.051042 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:14.051005 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:36:14.051316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:14.051249 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" containerID="cri-o://239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4" gracePeriod=30 Oct 30 03:36:19.313472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.313406 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:36:19.441105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441073 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441217 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441264 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441292 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441330 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:19.441470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441363 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441420 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441480 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441531 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441570 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:19.441642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441633 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\") pod \"87174551-f9aa-4157-8122-2b6290981cfc\" (UID: \"87174551-f9aa-4157-8122-2b6290981cfc\") " Oct 30 03:36:19.441854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441682 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:19.441854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441753 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:19.441854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.441676 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:36:19.442103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.442067 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.442103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.442105 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87174551-f9aa-4157-8122-2b6290981cfc-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.442297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.442120 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.442297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.442133 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.442297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.442148 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.443500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.443475 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:19.443626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.443499 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:36:19.443626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.443612 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "kube-api-access-st7zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:36:19.443823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.443805 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token" pod "87174551-f9aa-4157-8122-2b6290981cfc" (UID: "87174551-f9aa-4157-8122-2b6290981cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:36:19.543659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.543620 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87174551-f9aa-4157-8122-2b6290981cfc-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.543659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.543652 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87174551-f9aa-4157-8122-2b6290981cfc-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.543659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.543661 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7zm\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-kube-api-access-st7zm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.543897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.543670 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87174551-f9aa-4157-8122-2b6290981cfc-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:19.921367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.921329 4560 generic.go:356] "Generic (PLEG): container finished" podID="87174551-f9aa-4157-8122-2b6290981cfc" containerID="239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4" exitCode=0 Oct 30 03:36:19.921367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.921373 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" Oct 30 03:36:19.921630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.921417 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" event={"ID":"87174551-f9aa-4157-8122-2b6290981cfc","Type":"ContainerDied","Data":"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4"} Oct 30 03:36:19.921630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.921471 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7" event={"ID":"87174551-f9aa-4157-8122-2b6290981cfc","Type":"ContainerDied","Data":"fb7655a06f4e53c1a61cc23304173287c803b204094a34fb815b456f280c7b39"} Oct 30 03:36:19.921630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.921496 4560 scope.go:122] "RemoveContainer" containerID="239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4" Oct 30 03:36:19.931531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.931496 4560 scope.go:122] "RemoveContainer" containerID="239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4" Oct 30 03:36:19.931845 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:19.931828 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4\": container with ID starting with 239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4 not found: ID does not exist" containerID="239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4" Oct 30 03:36:19.931923 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.931857 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4"} err="failed to get container status \"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4\": rpc error: code = NotFound desc = could not find container \"239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4\": container with ID starting with 239e05808d024f332d0eabaa78862ed31ead68711390bd28c095b649e81244f4 not found: ID does not exist" Oct 30 03:36:19.986938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.986909 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:36:19.999504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:19.999474 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-2c7g7"] Oct 30 03:36:20.152049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:20.152019 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87174551-f9aa-4157-8122-2b6290981cfc" path="/var/lib/kubelet/pods/87174551-f9aa-4157-8122-2b6290981cfc/volumes" Oct 30 03:36:21.550138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:21.550105 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:36:21.550567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:21.550349 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" containerID="cri-o://9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a" gracePeriod=30 Oct 30 03:36:24.027896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:24.027859 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj"] Oct 30 03:36:24.040455 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:24.040425 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990a35a-2m6wj"] Oct 30 03:36:24.152135 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:24.152100 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f4bf0c-3c6f-47a1-84ac-766125ed652a" path="/var/lib/kubelet/pods/19f4bf0c-3c6f-47a1-84ac-766125ed652a/volumes" Oct 30 03:36:26.816413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.816358 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:36:26.925812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.925772 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.925812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.925814 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.925862 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.925941 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.925990 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926017 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926066 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:26.926249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926141 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926216 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926458 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926250 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:26.926458 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926291 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\") pod \"8f93672c-456c-4549-819b-d09b89995cb0\" (UID: \"8f93672c-456c-4549-819b-d09b89995cb0\") " Oct 30 03:36:26.926575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926462 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:26.926575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926486 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:36:26.926672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926650 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:26.926901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926880 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:26.926945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926908 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:26.926945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926923 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f93672c-456c-4549-819b-d09b89995cb0-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:26.926945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926937 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:26.927036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.926949 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:26.928253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.928225 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:36:26.928357 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.928280 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "kube-api-access-nxrg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:36:26.928754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.928732 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:36:26.928833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.928751 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token" pod "8f93672c-456c-4549-819b-d09b89995cb0" (UID: "8f93672c-456c-4549-819b-d09b89995cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:36:26.962967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.962929 4560 generic.go:356] "Generic (PLEG): container finished" podID="8f93672c-456c-4549-819b-d09b89995cb0" containerID="9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a" exitCode=0 Oct 30 03:36:26.963147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.962979 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" Oct 30 03:36:26.963147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.963020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" event={"ID":"8f93672c-456c-4549-819b-d09b89995cb0","Type":"ContainerDied","Data":"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a"} Oct 30 03:36:26.963147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.963082 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v" event={"ID":"8f93672c-456c-4549-819b-d09b89995cb0","Type":"ContainerDied","Data":"675d7b31529fec87b9ef5f2c550a40ce8e40138e7f0501b983f135b5626c6d72"} Oct 30 03:36:26.963147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.963106 4560 scope.go:122] "RemoveContainer" containerID="9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a" Oct 30 03:36:26.973586 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.973568 4560 scope.go:122] "RemoveContainer" containerID="9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a" Oct 30 03:36:26.973848 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:26.973830 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a\": container with ID starting with 9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a not found: ID does not exist" containerID="9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a" Oct 30 03:36:26.973901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:26.973858 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a"} err="failed to get container status \"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a\": rpc error: code = NotFound desc = could not find container \"9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a\": container with ID starting with 9f39117fe3b7267456bae35bb57be3a191be45529f8c1fe653a9cdd4b8c9cb8a not found: ID does not exist" Oct 30 03:36:27.018972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.018942 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:36:27.022865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.022840 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fcd5v"] Oct 30 03:36:27.028369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.028334 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:27.028369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.028362 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f93672c-456c-4549-819b-d09b89995cb0-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:27.028579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.028378 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f93672c-456c-4549-819b-d09b89995cb0-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:27.028579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:27.028392 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrg8\" (UniqueName: \"kubernetes.io/projected/8f93672c-456c-4549-819b-d09b89995cb0-kube-api-access-nxrg8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:36:28.152676 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:28.152638 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f93672c-456c-4549-819b-d09b89995cb0" path="/var/lib/kubelet/pods/8f93672c-456c-4549-819b-d09b89995cb0/volumes" Oct 30 03:36:31.225563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.225511 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:36:31.226278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226252 4560 cpu_manager.go:414] "containerMap: removing container" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" Oct 30 03:36:31.226382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226279 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" Oct 30 03:36:31.226382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226295 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" Oct 30 03:36:31.226382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226303 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" Oct 30 03:36:31.226482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226434 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f93672c-456c-4549-819b-d09b89995cb0" containerName="istio-proxy" Oct 30 03:36:31.226482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.226445 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="87174551-f9aa-4157-8122-2b6290981cfc" containerName="istio-proxy" Oct 30 03:36:31.231383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.231359 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.248284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.248256 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:36:31.258212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.258169 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"kube-root-ca.crt\"" Oct 30 03:36:31.258407 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.258380 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"openshift-service-ca.crt\"" Oct 30 03:36:31.258931 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.258911 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"default-dockercfg-lcxtm\"" Oct 30 03:36:31.258999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.258981 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"auth-test-model-547a78-kserve-self-signed-certs\"" Oct 30 03:36:31.275594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275563 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.275753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275609 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.275753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275691 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.275753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275726 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkhr\" (UniqueName: \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.275753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275746 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.275900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.275790 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.376949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.376901 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.376964 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377007 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377035 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkhr\" (UniqueName: \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377055 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377077 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377346 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377452 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.377721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.377635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.379666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.379643 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.380056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.380033 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.396663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.396627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkhr\" (UniqueName: \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\") pod \"auth-test-model-547a78-kserve-6d9b94dd77-f7kp6\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.545200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.545111 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:31.852882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.852852 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:36:31.854189 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:36:31.854153 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd17fb968_cfe4_486c_9c89_5efd18129279.slice/crio-1f7aa592aa8df6048879b5051daa6ea74dc74c86cc453d261368f69c86ead545 WatchSource:0}: Error finding container 1f7aa592aa8df6048879b5051daa6ea74dc74c86cc453d261368f69c86ead545: Status 404 returned error can't find the container with id 1f7aa592aa8df6048879b5051daa6ea74dc74c86cc453d261368f69c86ead545 Oct 30 03:36:31.997058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.997025 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" event={"ID":"d17fb968-cfe4-486c-9c89-5efd18129279","Type":"ContainerStarted","Data":"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82"} Oct 30 03:36:31.997058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:31.997063 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" event={"ID":"d17fb968-cfe4-486c-9c89-5efd18129279","Type":"ContainerStarted","Data":"1f7aa592aa8df6048879b5051daa6ea74dc74c86cc453d261368f69c86ead545"} Oct 30 03:36:32.036356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:32.036295 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" podStartSLOduration=1.036275764 podStartE2EDuration="1.036275764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:32.035468494 +0000 UTC m=+2020.020588364" watchObservedRunningTime="2026-10-30 03:36:32.036275764 +0000 UTC m=+2020.021395654" Oct 30 03:36:32.545496 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:32.545452 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:41.031401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.031317 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:36:41.036036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.036008 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.065883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.065854 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"openshift-service-ca.crt\"" Oct 30 03:36:41.066083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.065836 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"auth-test-model-7d99a4-kserve-self-signed-certs\"" Oct 30 03:36:41.066083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.065881 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"default-dockercfg-kftdj\"" Oct 30 03:36:41.066234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.065926 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"kube-root-ca.crt\"" Oct 30 03:36:41.068489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.068468 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:36:41.089801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.089761 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4249\" (UniqueName: \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.089958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.089872 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.089958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.089924 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.089958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.089949 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.090118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.089992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.090118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.090027 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191085 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191040 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191147 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4249\" (UniqueName: \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191261 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191701 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.191887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191856 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.192002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.191900 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.192251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.192226 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.192757 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.192365 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.194229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.194204 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.195115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.195089 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.210881 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.210856 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4249\" (UniqueName: \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\") pod \"auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.355004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.354907 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:41.545585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.545549 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:41.557972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.557947 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:41.658813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:41.658787 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:36:41.660294 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:36:41.660271 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbada471d_0075_4300_a439_e156a6233d1b.slice/crio-7f1526cf7b08924361e53aa8c7cf89679c092f84316ac50e8374cf09ac98f4a3 WatchSource:0}: Error finding container 7f1526cf7b08924361e53aa8c7cf89679c092f84316ac50e8374cf09ac98f4a3: Status 404 returned error can't find the container with id 7f1526cf7b08924361e53aa8c7cf89679c092f84316ac50e8374cf09ac98f4a3 Oct 30 03:36:41.750879 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:36:41.750844 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:36:42.058894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:42.058853 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" event={"ID":"bada471d-0075-4300-a439-e156a6233d1b","Type":"ContainerStarted","Data":"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d"} Oct 30 03:36:42.058894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:42.058896 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" event={"ID":"bada471d-0075-4300-a439-e156a6233d1b","Type":"ContainerStarted","Data":"7f1526cf7b08924361e53aa8c7cf89679c092f84316ac50e8374cf09ac98f4a3"} Oct 30 03:36:42.070380 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:42.070348 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:36:42.100005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:42.099952 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" podStartSLOduration=1.099939993 podStartE2EDuration="1.099939993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:42.098984917 +0000 UTC m=+2030.084104810" watchObservedRunningTime="2026-10-30 03:36:42.099939993 +0000 UTC m=+2030.085059880" Oct 30 03:36:42.355155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:42.355076 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:51.355876 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:51.355809 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:51.368309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:51.368275 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:36:52.129274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:36:52.129245 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:37:01.649358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.649321 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:37:01.654974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.654947 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.682974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.682943 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"default-dockercfg-tvn88\"" Oct 30 03:37:01.683168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.683095 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"kube-root-ca.crt\"" Oct 30 03:37:01.683168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.683096 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"openshift-service-ca.crt\"" Oct 30 03:37:01.683168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.683102 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"auth-test-model-852107-kserve-self-signed-certs\"" Oct 30 03:37:01.685366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.685345 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:37:01.719344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719288 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxbf\" (UniqueName: \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.719548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719419 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.719548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719476 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.719700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719609 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.719700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719634 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.719700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.719653 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.820909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.820871 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxbf\" (UniqueName: \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.820936 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.820979 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821068 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821096 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821570 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821575 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.821696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.821624 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.823638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.823614 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.824187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.824164 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.840865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.840830 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxbf\" (UniqueName: \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\") pod \"auth-test-model-852107-kserve-7fc4c79ffc-pbl8d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:01.969119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:01.969082 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:02.284780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:02.284752 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:37:02.286736 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:37:02.286709 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod656b80a9_24a7_4a50_ac32_df076598b41d.slice/crio-8c4e93cadfec5510dc00cf492bd5749a893f6e894e43e812454923ad54904456 WatchSource:0}: Error finding container 8c4e93cadfec5510dc00cf492bd5749a893f6e894e43e812454923ad54904456: Status 404 returned error can't find the container with id 8c4e93cadfec5510dc00cf492bd5749a893f6e894e43e812454923ad54904456 Oct 30 03:37:03.185951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:03.185909 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" event={"ID":"656b80a9-24a7-4a50-ac32-df076598b41d","Type":"ContainerStarted","Data":"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3"} Oct 30 03:37:03.186354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:03.185963 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" event={"ID":"656b80a9-24a7-4a50-ac32-df076598b41d","Type":"ContainerStarted","Data":"8c4e93cadfec5510dc00cf492bd5749a893f6e894e43e812454923ad54904456"} Oct 30 03:37:03.225023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:03.224973 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" podStartSLOduration=2.224959974 podStartE2EDuration="2.224959974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:03.224588736 +0000 UTC m=+2051.209708624" watchObservedRunningTime="2026-10-30 03:37:03.224959974 +0000 UTC m=+2051.210079907" Oct 30 03:37:03.970258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:03.970222 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:11.749041 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:37:11.749006 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:37:11.969733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:11.969700 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:11.982269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:11.982244 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:12.245698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:12.245666 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:37:39.372533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.372494 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:37:39.376794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.376771 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.408997 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.408964 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:37:39.418270 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.418228 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-75c2b0\"" Oct 30 03:37:39.439334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.439294 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.439668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.439631 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4nw\" (UniqueName: \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.471263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.471233 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-75c2b0-dockercfg-8t4vf\"" Oct 30 03:37:39.541061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.541026 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.541228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.541067 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4nw\" (UniqueName: \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.541824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.541802 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.561686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.561655 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4nw\" (UniqueName: \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\") pod \"payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:39.691179 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:39.691102 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:40.001330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:40.001294 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:37:40.002654 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:37:40.002626 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a905d69_b772_47de_a972_8c152bff1017.slice/crio-84c35baa5e29b65919ab2cb3b1e1ef19f48c36fbf5d178bb78825f99619c9f74 WatchSource:0}: Error finding container 84c35baa5e29b65919ab2cb3b1e1ef19f48c36fbf5d178bb78825f99619c9f74: Status 404 returned error can't find the container with id 84c35baa5e29b65919ab2cb3b1e1ef19f48c36fbf5d178bb78825f99619c9f74 Oct 30 03:37:40.396490 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:40.396401 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" event={"ID":"6a905d69-b772-47de-a972-8c152bff1017","Type":"ContainerStarted","Data":"5c8a1720868f02eba902f9f895b3fbbf3634f9480f6df2e85736794cb03ce6b7"} Oct 30 03:37:40.396902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:40.396529 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:40.396902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:40.396554 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" event={"ID":"6a905d69-b772-47de-a972-8c152bff1017","Type":"ContainerStarted","Data":"84c35baa5e29b65919ab2cb3b1e1ef19f48c36fbf5d178bb78825f99619c9f74"} Oct 30 03:37:40.435504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:40.435450 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" podStartSLOduration=1.4354255710000001 podStartE2EDuration="1.435425571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:40.43531785 +0000 UTC m=+2088.420437737" watchObservedRunningTime="2026-10-30 03:37:40.435425571 +0000 UTC m=+2088.420545461" Oct 30 03:37:41.749587 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:37:41.749551 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:37:51.405048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:51.405012 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:37:52.795066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:52.795030 4560 scope.go:122] "RemoveContainer" containerID="d9ee0f22a1cbefaaec5d5d6898d0b5ccbdedd5de7b45645f6153d1a2ccbb1da4" Oct 30 03:37:52.804622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:52.804597 4560 scope.go:122] "RemoveContainer" containerID="b7423e993f07e15473a8faeec8a803632add28a4fbf45febf80dcf27937aa260" Oct 30 03:37:57.497488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.497452 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:37:57.502688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.502672 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.521366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.521335 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-75c2b0-serving-cert\"" Oct 30 03:37:57.522965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.522940 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:37:57.547658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.547624 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7hf\" (UniqueName: \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.547806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.547740 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.547806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.547763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.547922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.547902 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.648811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.648773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7hf\" (UniqueName: \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.649013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.648839 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.649013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.648859 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.649013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.648921 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.649665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.649635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.651577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.651554 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.651924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.651902 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.668139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.668108 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7hf\" (UniqueName: \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\") pod \"maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:57.816238 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:57.816141 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:58.124004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:58.123918 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:37:58.127680 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:37:58.127652 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28037df_44c0_4974_8960_4198e8b5bc93.slice/crio-3b1490102c043fb1c3a4f050b441bcdb0b9430d9ba81ce1dd60d5880978c4cbe WatchSource:0}: Error finding container 3b1490102c043fb1c3a4f050b441bcdb0b9430d9ba81ce1dd60d5880978c4cbe: Status 404 returned error can't find the container with id 3b1490102c043fb1c3a4f050b441bcdb0b9430d9ba81ce1dd60d5880978c4cbe Oct 30 03:37:58.504049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:58.504019 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" event={"ID":"b28037df-44c0-4974-8960-4198e8b5bc93","Type":"ContainerStarted","Data":"3b1490102c043fb1c3a4f050b441bcdb0b9430d9ba81ce1dd60d5880978c4cbe"} Oct 30 03:37:59.511168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:59.511133 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" event={"ID":"b28037df-44c0-4974-8960-4198e8b5bc93","Type":"ContainerStarted","Data":"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e"} Oct 30 03:37:59.511684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:59.511292 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:37:59.568098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:37:59.568035 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" podStartSLOduration=2.193367841 podStartE2EDuration="2.56802069s" totalImagesPullingTime="374.652849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:59.565603306 +0000 UTC m=+2107.550723195" watchObservedRunningTime="2026-10-30 03:37:59.56802069 +0000 UTC m=+2107.553140578" Oct 30 03:38:05.521915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:38:05.521833 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:38:11.749644 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:38:11.749599 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:38:41.753077 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:38:41.753042 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:39:04.630686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:04.630604 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:39:11.752064 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:11.752021 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:39:14.832274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:14.832240 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:39:19.657021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.656953 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerName="main" containerID="cri-o://8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799" gracePeriod=45 Oct 30 03:39:19.832834 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.832763 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" podUID="6a905d69-b772-47de-a972-8c152bff1017" containerName="payload-pre-processing" containerID="cri-o://5c8a1720868f02eba902f9f895b3fbbf3634f9480f6df2e85736794cb03ce6b7" gracePeriod=25 Oct 30 03:39:19.963999 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.963970 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:39:19.971991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.971953 4560 generic.go:356] "Generic (PLEG): container finished" podID="6a905d69-b772-47de-a972-8c152bff1017" containerID="5c8a1720868f02eba902f9f895b3fbbf3634f9480f6df2e85736794cb03ce6b7" exitCode=0 Oct 30 03:39:19.972139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.972031 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" event={"ID":"6a905d69-b772-47de-a972-8c152bff1017","Type":"ContainerDied","Data":"5c8a1720868f02eba902f9f895b3fbbf3634f9480f6df2e85736794cb03ce6b7"} Oct 30 03:39:19.973889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.973862 4560 generic.go:356] "Generic (PLEG): container finished" podID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerID="8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799" exitCode=0 Oct 30 03:39:19.974023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.973913 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" Oct 30 03:39:19.974023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.973950 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" event={"ID":"d5c9c5c3-f5ae-479b-9180-44c8c238812b","Type":"ContainerDied","Data":"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799"} Oct 30 03:39:19.974023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.973996 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" event={"ID":"d5c9c5c3-f5ae-479b-9180-44c8c238812b","Type":"ContainerDied","Data":"b283171a1d37a930abd6f09a87fc0aa309e9861b3447833286241afec238ed3b"} Oct 30 03:39:19.974023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.974019 4560 scope.go:122] "RemoveContainer" containerID="8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799" Oct 30 03:39:19.985141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.984542 4560 scope.go:122] "RemoveContainer" containerID="8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799" Oct 30 03:39:19.985204 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:19.985145 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799\": container with ID starting with 8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799 not found: ID does not exist" containerID="8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799" Oct 30 03:39:19.985204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.985177 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799"} err="failed to get container status \"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799\": rpc error: code = NotFound desc = could not find container \"8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799\": container with ID starting with 8a999eb0878c516569b6b7f016bf780744555be0be62c634cdfa181295b2d799 not found: ID does not exist" Oct 30 03:39:19.996121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996095 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996248 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996307 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996318 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:19.996450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996368 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996419 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\" (UniqueName: \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996485 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\") pod \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\" (UID: \"d5c9c5c3-f5ae-479b-9180-44c8c238812b\") " Oct 30 03:39:19.996585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996484 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:19.996752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.996726 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:19.997042 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.997024 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:19.997114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.997046 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:19.997114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.997062 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:19.998814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.998791 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:39:19.998902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.998799 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:19.998902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:19.998816 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7" pod "d5c9c5c3-f5ae-479b-9180-44c8c238812b" (UID: "d5c9c5c3-f5ae-479b-9180-44c8c238812b"). InnerVolumeSpecName "kube-api-access-87rq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:20.083176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.083150 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:39:20.097879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.097850 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\" (UniqueName: \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\") pod \"6a905d69-b772-47de-a972-8c152bff1017\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " Oct 30 03:39:20.098007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.097898 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\") pod \"6a905d69-b772-47de-a972-8c152bff1017\" (UID: \"6a905d69-b772-47de-a972-8c152bff1017\") " Oct 30 03:39:20.098405 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.098378 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume" pod "6a905d69-b772-47de-a972-8c152bff1017" (UID: "6a905d69-b772-47de-a972-8c152bff1017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:39:20.098569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.098455 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c9c5c3-f5ae-479b-9180-44c8c238812b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.098569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.098470 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rq7\" (UniqueName: \"kubernetes.io/projected/d5c9c5c3-f5ae-479b-9180-44c8c238812b-kube-api-access-87rq7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.098569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.098479 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c9c5c3-f5ae-479b-9180-44c8c238812b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.100070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.100044 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw" pod "6a905d69-b772-47de-a972-8c152bff1017" (UID: "6a905d69-b772-47de-a972-8c152bff1017"). InnerVolumeSpecName "kube-api-access-9t4nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:20.199970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.199945 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4nw\" (UniqueName: \"kubernetes.io/projected/6a905d69-b772-47de-a972-8c152bff1017-kube-api-access-9t4nw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.199970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.199969 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a905d69-b772-47de-a972-8c152bff1017-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.312419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.312385 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:39:20.317813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.317784 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5"] Oct 30 03:39:20.837265 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.837226 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-da4553/auth-test-model-da4553-kserve-64f5cc7d5-wxfz5" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 03:39:20.982663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.982628 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" event={"ID":"6a905d69-b772-47de-a972-8c152bff1017","Type":"ContainerDied","Data":"84c35baa5e29b65919ab2cb3b1e1ef19f48c36fbf5d178bb78825f99619c9f74"} Oct 30 03:39:20.982663 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.982649 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj" Oct 30 03:39:20.982880 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:20.982687 4560 scope.go:122] "RemoveContainer" containerID="5c8a1720868f02eba902f9f895b3fbbf3634f9480f6df2e85736794cb03ce6b7" Oct 30 03:39:21.039148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:21.039107 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:39:21.039148 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:21.039157 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75c2b0-7fd559b87f-n95fj"] Oct 30 03:39:22.159016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:22.158703 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a905d69-b772-47de-a972-8c152bff1017" path="/var/lib/kubelet/pods/6a905d69-b772-47de-a972-8c152bff1017/volumes" Oct 30 03:39:22.160593 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:22.160568 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" path="/var/lib/kubelet/pods/d5c9c5c3-f5ae-479b-9180-44c8c238812b/volumes" Oct 30 03:39:28.288618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.288577 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:39:28.289031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.288792 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" podUID="b28037df-44c0-4974-8960-4198e8b5bc93" containerName="maas-api" containerID="cri-o://5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e" gracePeriod=30 Oct 30 03:39:28.577231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.577189 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:39:28.698291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.698257 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\") pod \"b28037df-44c0-4974-8960-4198e8b5bc93\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " Oct 30 03:39:28.698469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.698393 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\") pod \"b28037df-44c0-4974-8960-4198e8b5bc93\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " Oct 30 03:39:28.698469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.698462 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\") pod \"b28037df-44c0-4974-8960-4198e8b5bc93\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " Oct 30 03:39:28.698648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.698627 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\" (UniqueName: \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\") pod \"b28037df-44c0-4974-8960-4198e8b5bc93\" (UID: \"b28037df-44c0-4974-8960-4198e8b5bc93\") " Oct 30 03:39:28.698860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.698833 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca" pod "b28037df-44c0-4974-8960-4198e8b5bc93" (UID: "b28037df-44c0-4974-8960-4198e8b5bc93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:39:28.699111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.699094 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28037df-44c0-4974-8960-4198e8b5bc93-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:28.700404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.700379 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls" pod "b28037df-44c0-4974-8960-4198e8b5bc93" (UID: "b28037df-44c0-4974-8960-4198e8b5bc93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:39:28.700509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.700494 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs" pod "b28037df-44c0-4974-8960-4198e8b5bc93" (UID: "b28037df-44c0-4974-8960-4198e8b5bc93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:39:28.700690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.700673 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf" pod "b28037df-44c0-4974-8960-4198e8b5bc93" (UID: "b28037df-44c0-4974-8960-4198e8b5bc93"). InnerVolumeSpecName "kube-api-access-9r7hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:28.800633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.800513 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7hf\" (UniqueName: \"kubernetes.io/projected/b28037df-44c0-4974-8960-4198e8b5bc93-kube-api-access-9r7hf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:28.800633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.800575 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:28.800633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.800591 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28037df-44c0-4974-8960-4198e8b5bc93-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:28.935196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:28.935159 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:39:29.038102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.038068 4560 generic.go:356] "Generic (PLEG): container finished" podID="b28037df-44c0-4974-8960-4198e8b5bc93" containerID="5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e" exitCode=0 Oct 30 03:39:29.038290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.038127 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" Oct 30 03:39:29.038290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.038153 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" event={"ID":"b28037df-44c0-4974-8960-4198e8b5bc93","Type":"ContainerDied","Data":"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e"} Oct 30 03:39:29.038290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.038220 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr" event={"ID":"b28037df-44c0-4974-8960-4198e8b5bc93","Type":"ContainerDied","Data":"3b1490102c043fb1c3a4f050b441bcdb0b9430d9ba81ce1dd60d5880978c4cbe"} Oct 30 03:39:29.038290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.038253 4560 scope.go:122] "RemoveContainer" containerID="5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e" Oct 30 03:39:29.048656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.048635 4560 scope.go:122] "RemoveContainer" containerID="5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e" Oct 30 03:39:29.048939 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:29.048917 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e\": container with ID starting with 5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e not found: ID does not exist" containerID="5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e" Oct 30 03:39:29.049009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.048951 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e"} err="failed to get container status \"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e\": rpc error: code = NotFound desc = could not find container \"5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e\": container with ID starting with 5212089f52b3848559e15033b12fa0de6dc75bbb4ecea646532f108472294c8e not found: ID does not exist" Oct 30 03:39:29.090662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.090571 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:39:29.095783 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:29.095748 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75c2b0-7d785b9d8f-cgqmr"] Oct 30 03:39:30.152575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:30.152544 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28037df-44c0-4974-8960-4198e8b5bc93" path="/var/lib/kubelet/pods/b28037df-44c0-4974-8960-4198e8b5bc93/volumes" Oct 30 03:39:32.405701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.405662 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:39:32.667285 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667201 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-211a06"] Oct 30 03:39:32.667701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667687 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerName="main" Oct 30 03:39:32.667763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667702 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerName="main" Oct 30 03:39:32.667763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667719 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6a905d69-b772-47de-a972-8c152bff1017" containerName="payload-pre-processing" Oct 30 03:39:32.667763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667725 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a905d69-b772-47de-a972-8c152bff1017" containerName="payload-pre-processing" Oct 30 03:39:32.667763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667737 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b28037df-44c0-4974-8960-4198e8b5bc93" containerName="maas-api" Oct 30 03:39:32.667763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667742 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28037df-44c0-4974-8960-4198e8b5bc93" containerName="maas-api" Oct 30 03:39:32.667941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667835 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c9c5c3-f5ae-479b-9180-44c8c238812b" containerName="main" Oct 30 03:39:32.667941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667846 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a905d69-b772-47de-a972-8c152bff1017" containerName="payload-pre-processing" Oct 30 03:39:32.667941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.667854 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28037df-44c0-4974-8960-4198e8b5bc93" containerName="maas-api" Oct 30 03:39:32.672494 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.672475 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:32.687510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.687473 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 03:39:32.689121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.689098 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-211a06"] Oct 30 03:39:32.744894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.744863 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dn7\" (UniqueName: \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\") pod \"test-curl-7573-211a06\" (UID: \"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49\") " pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:32.845917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.845878 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dn7\" (UniqueName: \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\") pod \"test-curl-7573-211a06\" (UID: \"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49\") " pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:32.864942 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.864918 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dn7\" (UniqueName: \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\") pod \"test-curl-7573-211a06\" (UID: \"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49\") " pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:32.984250 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:32.984214 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:33.285581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:33.285550 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-211a06"] Oct 30 03:39:33.286848 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:33.286820 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62c8c5f1_a9ee_485b_b1f5_bf65ebb5fa49.slice/crio-9b874a83d77f7b150a4d62496fa429c30be1ec40cc3174325c629f2616cfefbd WatchSource:0}: Error finding container 9b874a83d77f7b150a4d62496fa429c30be1ec40cc3174325c629f2616cfefbd: Status 404 returned error can't find the container with id 9b874a83d77f7b150a4d62496fa429c30be1ec40cc3174325c629f2616cfefbd Oct 30 03:39:34.069777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:34.069694 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-211a06" event={"ID":"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49","Type":"ContainerStarted","Data":"d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73"} Oct 30 03:39:34.069777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:34.069734 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-211a06" event={"ID":"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49","Type":"ContainerStarted","Data":"9b874a83d77f7b150a4d62496fa429c30be1ec40cc3174325c629f2616cfefbd"} Oct 30 03:39:34.117321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:34.117270 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7573-211a06" podStartSLOduration=2.117255073 podStartE2EDuration="2.117255073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:34.115629689 +0000 UTC m=+2202.100749579" watchObservedRunningTime="2026-10-30 03:39:34.117255073 +0000 UTC m=+2202.102374965" Oct 30 03:39:34.960918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:34.960883 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-211a06"] Oct 30 03:39:34.964748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:34.964715 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-211a06"] Oct 30 03:39:36.080396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:36.080356 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7573-211a06" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" containerName="test-curl-7573-211a06" containerID="cri-o://d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73" gracePeriod=2 Oct 30 03:39:36.094074 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:36.094033 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-211a06\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:38.317049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:38.317020 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:38.409349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:38.409258 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\" (UniqueName: \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\") pod \"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49\" (UID: \"62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49\") " Oct 30 03:39:38.411666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:38.411630 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7" pod "62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" (UID: "62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49"). InnerVolumeSpecName "kube-api-access-r6dn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:38.511333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:38.511289 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6dn7\" (UniqueName: \"kubernetes.io/projected/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49-kube-api-access-r6dn7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:39.101386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:39.101350 4560 generic.go:356] "Generic (PLEG): container finished" podID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" containerID="d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73" exitCode=137 Oct 30 03:39:39.101606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:39.101397 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:39.101606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:39.101412 4560 scope.go:122] "RemoveContainer" containerID="d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73" Oct 30 03:39:39.112227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:39.112211 4560 scope.go:122] "RemoveContainer" containerID="d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73" Oct 30 03:39:39.112475 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:39.112452 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73\": container with ID starting with d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73 not found: ID does not exist" containerID="d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73" Oct 30 03:39:39.112587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:39.112486 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73"} err="failed to get container status \"d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73\": rpc error: code = NotFound desc = could not find container \"d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73\": container with ID starting with d3054936ac401932eca423b41195678ab503e624ac24000a3d4f5e86ce924a73 not found: ID does not exist" Oct 30 03:39:39.124948 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:39.124920 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-211a06\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" pod="opendatahub/test-curl-7573-211a06" Oct 30 03:39:40.152032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:40.151999 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" path="/var/lib/kubelet/pods/62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49/volumes" Oct 30 03:39:41.750567 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:41.750509 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:39:42.706377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.706346 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 03:39:42.707075 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.707037 4560 cpu_manager.go:414] "containerMap: removing container" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" containerName="test-curl-7573-211a06" Oct 30 03:39:42.707221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.707078 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" containerName="test-curl-7573-211a06" Oct 30 03:39:42.707298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.707240 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c8c5f1-a9ee-485b-b1f5-bf65ebb5fa49" containerName="test-curl-7573-211a06" Oct 30 03:39:42.712326 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.712298 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.755322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.755280 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 03:39:42.764752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.764728 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9d8692\"" Oct 30 03:39:42.765314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.765290 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9d8692-dockercfg-8fxb4\"" Oct 30 03:39:42.793582 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.793550 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 03:39:42.797721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.797702 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:42.823547 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.823506 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 03:39:42.862778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.862743 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.863020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.862992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5t4\" (UniqueName: \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.964320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.964229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5t4\" (UniqueName: \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.964320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.964307 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9bq\" (UniqueName: \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:42.964598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.964341 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.964598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.964495 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:42.965132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.965108 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:42.983597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:42.983570 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5t4\" (UniqueName: \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\") pod \"payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:43.029240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.029202 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:43.065412 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.065379 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9bq\" (UniqueName: \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:43.065606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.065480 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:43.066214 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.066192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:43.085311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.085280 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9bq\" (UniqueName: \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\") pod \"payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:43.114689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.113044 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:43.351826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.351800 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 03:39:43.353020 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:43.352996 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc765941d_9664_4e4b_b039_2b255c1ade47.slice/crio-3af3d5d5f917f4e1a741b860d91e5027fd7fc7f0b3b32addcade7e0c8ffb00ed WatchSource:0}: Error finding container 3af3d5d5f917f4e1a741b860d91e5027fd7fc7f0b3b32addcade7e0c8ffb00ed: Status 404 returned error can't find the container with id 3af3d5d5f917f4e1a741b860d91e5027fd7fc7f0b3b32addcade7e0c8ffb00ed Oct 30 03:39:43.428114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.428087 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 03:39:43.429747 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:43.429715 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29d3c12_21bb_43f6_8bdf_33791fb683e5.slice/crio-9c1957dda8adda2b10c2279b2cc117e2329eea0f6fed240c517212b0f5ac4a95 WatchSource:0}: Error finding container 9c1957dda8adda2b10c2279b2cc117e2329eea0f6fed240c517212b0f5ac4a95: Status 404 returned error can't find the container with id 9c1957dda8adda2b10c2279b2cc117e2329eea0f6fed240c517212b0f5ac4a95 Oct 30 03:39:43.956351 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:43.956306 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" podUID="e2e1874d-4482-42d2-b054-93e7ebdace49" containerName="main" containerID="cri-o://69f2a6151a6e2790a7b1faf2d2cb044902ce1107865cdbd30a1fc8261b30c86d" gracePeriod=45 Oct 30 03:39:44.156423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.156396 4560 generic.go:356] "Generic (PLEG): container finished" podID="e2e1874d-4482-42d2-b054-93e7ebdace49" containerID="69f2a6151a6e2790a7b1faf2d2cb044902ce1107865cdbd30a1fc8261b30c86d" exitCode=0 Oct 30 03:39:44.157600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.157576 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:44.157734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.157631 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" event={"ID":"c765941d-9664-4e4b-b039-2b255c1ade47","Type":"ContainerStarted","Data":"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741"} Oct 30 03:39:44.157734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.157657 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" event={"ID":"c765941d-9664-4e4b-b039-2b255c1ade47","Type":"ContainerStarted","Data":"3af3d5d5f917f4e1a741b860d91e5027fd7fc7f0b3b32addcade7e0c8ffb00ed"} Oct 30 03:39:44.157734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.157679 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" event={"ID":"e2e1874d-4482-42d2-b054-93e7ebdace49","Type":"ContainerDied","Data":"69f2a6151a6e2790a7b1faf2d2cb044902ce1107865cdbd30a1fc8261b30c86d"} Oct 30 03:39:44.158187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.158163 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" event={"ID":"f29d3c12-21bb-43f6-8bdf-33791fb683e5","Type":"ContainerStarted","Data":"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c"} Oct 30 03:39:44.158279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.158208 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" event={"ID":"f29d3c12-21bb-43f6-8bdf-33791fb683e5","Type":"ContainerStarted","Data":"9c1957dda8adda2b10c2279b2cc117e2329eea0f6fed240c517212b0f5ac4a95"} Oct 30 03:39:44.158279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.158265 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:44.194136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.194082 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" podStartSLOduration=2.194064218 podStartE2EDuration="2.194064218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:44.193887118 +0000 UTC m=+2212.179007007" watchObservedRunningTime="2026-10-30 03:39:44.194064218 +0000 UTC m=+2212.179184106" Oct 30 03:39:44.232309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.232283 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:39:44.235472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.235421 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" podStartSLOduration=2.235406455 podStartE2EDuration="2.235406455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:44.234115593 +0000 UTC m=+2212.219235480" watchObservedRunningTime="2026-10-30 03:39:44.235406455 +0000 UTC m=+2212.220526344" Oct 30 03:39:44.379187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379153 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379393 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379247 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\" (UniqueName: \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379393 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379296 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379393 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379364 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379399 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379486 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\") pod \"e2e1874d-4482-42d2-b054-93e7ebdace49\" (UID: \"e2e1874d-4482-42d2-b054-93e7ebdace49\") " Oct 30 03:39:44.379628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379493 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:44.379790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379738 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:44.379790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:44.379995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379963 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:44.379995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.379996 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:44.380186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.380010 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:44.381680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.381656 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:39:44.381783 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.381658 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "kube-api-access-jfb6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:44.381783 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.381708 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm" pod "e2e1874d-4482-42d2-b054-93e7ebdace49" (UID: "e2e1874d-4482-42d2-b054-93e7ebdace49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:44.480966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.480893 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfb6x\" (UniqueName: \"kubernetes.io/projected/e2e1874d-4482-42d2-b054-93e7ebdace49-kube-api-access-jfb6x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:44.480966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.480920 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e1874d-4482-42d2-b054-93e7ebdace49-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:44.480966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:44.480930 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e1874d-4482-42d2-b054-93e7ebdace49-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:45.164814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:45.164780 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" Oct 30 03:39:45.164814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:45.164801 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz" event={"ID":"e2e1874d-4482-42d2-b054-93e7ebdace49","Type":"ContainerDied","Data":"8f7772ec9993b35bad823c2c15270ec1db6962dda44d33763f81d044615230e0"} Oct 30 03:39:45.165369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:45.164848 4560 scope.go:122] "RemoveContainer" containerID="69f2a6151a6e2790a7b1faf2d2cb044902ce1107865cdbd30a1fc8261b30c86d" Oct 30 03:39:45.216394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:45.216354 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:39:45.228704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:45.228672 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/auth-test-model-1a1b45-kserve-6488fc8fbf-6gkxz"] Oct 30 03:39:46.154210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:46.154170 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e1874d-4482-42d2-b054-93e7ebdace49" path="/var/lib/kubelet/pods/e2e1874d-4482-42d2-b054-93e7ebdace49/volumes" Oct 30 03:39:47.427610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.427552 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" podUID="d17fb968-cfe4-486c-9c89-5efd18129279" containerName="main" containerID="cri-o://569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82" gracePeriod=45 Oct 30 03:39:47.577945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.577903 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 03:39:47.578688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.578668 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e2e1874d-4482-42d2-b054-93e7ebdace49" containerName="main" Oct 30 03:39:47.578741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.578692 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e1874d-4482-42d2-b054-93e7ebdace49" containerName="main" Oct 30 03:39:47.578892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.578880 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e1874d-4482-42d2-b054-93e7ebdace49" containerName="main" Oct 30 03:39:47.582658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.582638 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.608548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.608488 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 03:39:47.616644 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.616621 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-689bcf-serving-cert\"" Oct 30 03:39:47.713444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.713416 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.713592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.713481 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.713659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.713598 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.713707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.713678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnqk\" (UniqueName: \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.733308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.733283 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:39:47.815104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.815070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.815303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.815166 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.815303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.815201 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.815303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.815248 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnqk\" (UniqueName: \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.815986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.815961 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.817625 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.817602 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.817729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.817710 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.835939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.835905 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnqk\" (UniqueName: \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\") pod \"maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.896994 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.896952 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:47.915855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.915820 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.915976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.915876 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.915976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.915931 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.915976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.915965 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.916134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916042 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\" (UniqueName: \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.916134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916068 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:47.916134 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916103 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\" (UniqueName: \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\") pod \"d17fb968-cfe4-486c-9c89-5efd18129279\" (UID: \"d17fb968-cfe4-486c-9c89-5efd18129279\") " Oct 30 03:39:47.916290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916241 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:47.916290 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916278 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:47.916633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916611 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:47.916731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916637 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:47.916731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.916652 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:47.918709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.918681 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:39:47.918817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.918758 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:39:47.918817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:47.918772 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr" pod "d17fb968-cfe4-486c-9c89-5efd18129279" (UID: "d17fb968-cfe4-486c-9c89-5efd18129279"). InnerVolumeSpecName "kube-api-access-6vkhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:48.017349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.017264 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkhr\" (UniqueName: \"kubernetes.io/projected/d17fb968-cfe4-486c-9c89-5efd18129279-kube-api-access-6vkhr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:48.017349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.017299 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17fb968-cfe4-486c-9c89-5efd18129279-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:48.017349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.017312 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17fb968-cfe4-486c-9c89-5efd18129279-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:39:48.184039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.184005 4560 generic.go:356] "Generic (PLEG): container finished" podID="d17fb968-cfe4-486c-9c89-5efd18129279" containerID="569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82" exitCode=0 Oct 30 03:39:48.184192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.184085 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" event={"ID":"d17fb968-cfe4-486c-9c89-5efd18129279","Type":"ContainerDied","Data":"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82"} Oct 30 03:39:48.184192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.184112 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" Oct 30 03:39:48.184192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.184128 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6" event={"ID":"d17fb968-cfe4-486c-9c89-5efd18129279","Type":"ContainerDied","Data":"1f7aa592aa8df6048879b5051daa6ea74dc74c86cc453d261368f69c86ead545"} Oct 30 03:39:48.184192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.184145 4560 scope.go:122] "RemoveContainer" containerID="569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82" Oct 30 03:39:48.193488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.193466 4560 scope.go:122] "RemoveContainer" containerID="569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82" Oct 30 03:39:48.193867 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:39:48.193840 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82\": container with ID starting with 569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82 not found: ID does not exist" containerID="569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82" Oct 30 03:39:48.193938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.193869 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82"} err="failed to get container status \"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82\": rpc error: code = NotFound desc = could not find container \"569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82\": container with ID starting with 569991934b20b9c2df39ff0618d5b6e27afbba0c8960b63de9aee02e66abab82 not found: ID does not exist" Oct 30 03:39:48.211848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.211815 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 03:39:48.215471 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:48.215447 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c984fe5_943a_403a_bd4f_8e467a9c7bb8.slice/crio-48148c508dd4a1a8cf86364a3cdc98c238438182e7a38ef9b1cf7fc3e4627110 WatchSource:0}: Error finding container 48148c508dd4a1a8cf86364a3cdc98c238438182e7a38ef9b1cf7fc3e4627110: Status 404 returned error can't find the container with id 48148c508dd4a1a8cf86364a3cdc98c238438182e7a38ef9b1cf7fc3e4627110 Oct 30 03:39:48.237890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.237834 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:39:48.240094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:48.240067 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/auth-test-model-547a78-kserve-6d9b94dd77-f7kp6"] Oct 30 03:39:49.192752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:49.192719 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" event={"ID":"2c984fe5-943a-403a-bd4f-8e467a9c7bb8","Type":"ContainerStarted","Data":"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438"} Oct 30 03:39:49.193143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:49.192769 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" event={"ID":"2c984fe5-943a-403a-bd4f-8e467a9c7bb8","Type":"ContainerStarted","Data":"48148c508dd4a1a8cf86364a3cdc98c238438182e7a38ef9b1cf7fc3e4627110"} Oct 30 03:39:49.193143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:49.192799 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:49.235795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:49.235739 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" podStartSLOduration=1.8047604800000001 podStartE2EDuration="2.2357217s" totalImagesPullingTime="430.96122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:49.233807838 +0000 UTC m=+2217.218927725" watchObservedRunningTime="2026-10-30 03:39:49.2357217 +0000 UTC m=+2217.220841587" Oct 30 03:39:50.152124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:50.152094 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17fb968-cfe4-486c-9c89-5efd18129279" path="/var/lib/kubelet/pods/d17fb968-cfe4-486c-9c89-5efd18129279/volumes" Oct 30 03:39:52.351755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.351717 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:39:52.352577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.352544 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d17fb968-cfe4-486c-9c89-5efd18129279" containerName="main" Oct 30 03:39:52.352577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.352574 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17fb968-cfe4-486c-9c89-5efd18129279" containerName="main" Oct 30 03:39:52.352736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.352704 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17fb968-cfe4-486c-9c89-5efd18129279" containerName="main" Oct 30 03:39:52.356851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.356822 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.373895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.373859 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:39:52.383731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.383695 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-22aeb4\"/\"default-dockercfg-qxgwl\"" Oct 30 03:39:52.383731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.383722 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-22aeb4\"/\"noten-model-22aeb4-kserve-self-signed-certs\"" Oct 30 03:39:52.384597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.384576 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-22aeb4\"/\"openshift-service-ca.crt\"" Oct 30 03:39:52.384719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.384653 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-22aeb4\"/\"kube-root-ca.crt\"" Oct 30 03:39:52.459159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459127 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.459335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459190 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.459335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459290 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8l9p\" (UniqueName: \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.459420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459354 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.459420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459383 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.459420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.459405 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560381 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8l9p\" (UniqueName: \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560476 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560501 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560545 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560651 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.560832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.560708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.561105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.561079 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.561253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.561128 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.561253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.561188 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.563031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.563006 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.563434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.563398 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.582580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.582538 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8l9p\" (UniqueName: \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\") pod \"noten-model-22aeb4-kserve-5b677bc8fc-jhdgw\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.672456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.672363 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:52.979612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:52.979585 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:39:52.980952 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:52.980922 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369ace70_48d6_4106_b37e_21d5d7535237.slice/crio-bd616f4b2a8bbf7e48f0e5b9c25e533bb18735ea1c86d48974547a74773cd946 WatchSource:0}: Error finding container bd616f4b2a8bbf7e48f0e5b9c25e533bb18735ea1c86d48974547a74773cd946: Status 404 returned error can't find the container with id bd616f4b2a8bbf7e48f0e5b9c25e533bb18735ea1c86d48974547a74773cd946 Oct 30 03:39:53.219146 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:53.219100 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" event={"ID":"369ace70-48d6-4106-b37e-21d5d7535237","Type":"ContainerStarted","Data":"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73"} Oct 30 03:39:53.219146 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:53.219149 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" event={"ID":"369ace70-48d6-4106-b37e-21d5d7535237","Type":"ContainerStarted","Data":"bd616f4b2a8bbf7e48f0e5b9c25e533bb18735ea1c86d48974547a74773cd946"} Oct 30 03:39:53.259960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:53.259846 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" podStartSLOduration=1.259830673 podStartE2EDuration="1.259830673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:53.25766038 +0000 UTC m=+2221.242780286" watchObservedRunningTime="2026-10-30 03:39:53.259830673 +0000 UTC m=+2221.244950562" Oct 30 03:39:53.672763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:53.672682 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:39:55.168645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.168611 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 03:39:55.169019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.168772 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 03:39:55.203101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.203074 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 03:39:55.540153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.540117 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7570-9a214b"] Oct 30 03:39:55.544100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.544080 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:39:55.558653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.558624 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-9a214b"] Oct 30 03:39:55.558966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.558945 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 03:39:55.594105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.594070 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqst\" (UniqueName: \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\") pod \"test-curl-7570-9a214b\" (UID: \"9db9f352-a2f4-4cb1-a29f-c392d654ce83\") " pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:39:55.695324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.695288 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqst\" (UniqueName: \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\") pod \"test-curl-7570-9a214b\" (UID: \"9db9f352-a2f4-4cb1-a29f-c392d654ce83\") " pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:39:55.714161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.714125 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqst\" (UniqueName: \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\") pod \"test-curl-7570-9a214b\" (UID: \"9db9f352-a2f4-4cb1-a29f-c392d654ce83\") " pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:39:55.856413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:55.856332 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:39:56.155175 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:56.155147 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-9a214b"] Oct 30 03:39:56.156689 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:39:56.156657 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9db9f352_a2f4_4cb1_a29f_c392d654ce83.slice/crio-e1c106e936f73d5153a8582574a55c9cfcb9f9d14e7695a539c57d54a06125bb WatchSource:0}: Error finding container e1c106e936f73d5153a8582574a55c9cfcb9f9d14e7695a539c57d54a06125bb: Status 404 returned error can't find the container with id e1c106e936f73d5153a8582574a55c9cfcb9f9d14e7695a539c57d54a06125bb Oct 30 03:39:56.237409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:56.237377 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-9a214b" event={"ID":"9db9f352-a2f4-4cb1-a29f-c392d654ce83","Type":"ContainerStarted","Data":"f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e"} Oct 30 03:39:56.237794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:56.237427 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-9a214b" event={"ID":"9db9f352-a2f4-4cb1-a29f-c392d654ce83","Type":"ContainerStarted","Data":"e1c106e936f73d5153a8582574a55c9cfcb9f9d14e7695a539c57d54a06125bb"} Oct 30 03:39:56.275651 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:56.275601 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7570-9a214b" podStartSLOduration=1.275588043 podStartE2EDuration="1.275588043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:56.272947555 +0000 UTC m=+2224.258067451" watchObservedRunningTime="2026-10-30 03:39:56.275588043 +0000 UTC m=+2224.260707931" Oct 30 03:39:57.099799 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:57.099760 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7570-9a214b"] Oct 30 03:39:57.103963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:57.103926 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7570-9a214b"] Oct 30 03:39:58.248011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:39:58.247970 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7570-9a214b" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" containerName="test-curl-7570-9a214b" containerID="cri-o://f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e" gracePeriod=2 Oct 30 03:40:00.444114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:00.444087 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:40:00.547243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:00.547137 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:40:00.547755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:00.547728 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\" (UniqueName: \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\") pod \"9db9f352-a2f4-4cb1-a29f-c392d654ce83\" (UID: \"9db9f352-a2f4-4cb1-a29f-c392d654ce83\") " Oct 30 03:40:00.551855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:00.551402 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst" pod "9db9f352-a2f4-4cb1-a29f-c392d654ce83" (UID: "9db9f352-a2f4-4cb1-a29f-c392d654ce83"). InnerVolumeSpecName "kube-api-access-nbqst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:00.650972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:00.650921 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbqst\" (UniqueName: \"kubernetes.io/projected/9db9f352-a2f4-4cb1-a29f-c392d654ce83-kube-api-access-nbqst\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:01.267275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:01.267235 4560 generic.go:356] "Generic (PLEG): container finished" podID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" containerID="f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e" exitCode=137 Oct 30 03:40:01.267275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:01.267280 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:40:01.267534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:01.267338 4560 scope.go:122] "RemoveContainer" containerID="f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e" Oct 30 03:40:01.277007 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:01.276981 4560 scope.go:122] "RemoveContainer" containerID="f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e" Oct 30 03:40:01.277303 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:01.277280 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e\": container with ID starting with f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e not found: ID does not exist" containerID="f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e" Oct 30 03:40:01.277373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:01.277313 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e"} err="failed to get container status \"f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e\": rpc error: code = NotFound desc = could not find container \"f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e\": container with ID starting with f8fb5e6b4038b96cf99c0522fec6704a0bea0cd97d73ca682c26dfb94fc0887e not found: ID does not exist" Oct 30 03:40:01.290585 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:01.290554 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7570-9a214b\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:40:02.154722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.154680 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" path="/var/lib/kubelet/pods/9db9f352-a2f4-4cb1-a29f-c392d654ce83/volumes" Oct 30 03:40:02.162413 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:02.162383 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7570-9a214b\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" pod="opendatahub/test-curl-7570-9a214b" Oct 30 03:40:02.356271 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.356223 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 03:40:02.356941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.356916 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" containerName="test-curl-7570-9a214b" Oct 30 03:40:02.356941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.356937 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" containerName="test-curl-7570-9a214b" Oct 30 03:40:02.357112 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.357087 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db9f352-a2f4-4cb1-a29f-c392d654ce83" containerName="test-curl-7570-9a214b" Oct 30 03:40:02.361823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.361796 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.385114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.385074 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 03:40:02.389340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.389302 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9d8692-serving-cert\"" Oct 30 03:40:02.474147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.474097 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tsx\" (UniqueName: \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.474314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.474179 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.474314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.474209 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.474421 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.474330 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.575918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.575885 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.576113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.575940 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.576113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.576025 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.576230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.576202 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tsx\" (UniqueName: \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.576835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.576812 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.578597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.578567 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.578712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.578626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.595180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.595151 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tsx\" (UniqueName: \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\") pod \"maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.677170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.677134 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:02.978176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:02.978139 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 03:40:02.979696 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:02.979644 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4f3ca2_ae0c_4fe6_8fab_18df4f62ab99.slice/crio-82025d3ca8b97342f4503f64e78d0e9cecc0a06727baef9180f73d0369e8d7a1 WatchSource:0}: Error finding container 82025d3ca8b97342f4503f64e78d0e9cecc0a06727baef9180f73d0369e8d7a1: Status 404 returned error can't find the container with id 82025d3ca8b97342f4503f64e78d0e9cecc0a06727baef9180f73d0369e8d7a1 Oct 30 03:40:03.282512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:03.282419 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" event={"ID":"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99","Type":"ContainerStarted","Data":"82025d3ca8b97342f4503f64e78d0e9cecc0a06727baef9180f73d0369e8d7a1"} Oct 30 03:40:04.289497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:04.289459 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" event={"ID":"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99","Type":"ContainerStarted","Data":"6239449d142007b10d86864b33fb8b3c9267d2be1a07c8d2e21c7639d3ec8e00"} Oct 30 03:40:04.290037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:04.289551 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:04.331250 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:04.331193 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" podStartSLOduration=1.860706182 podStartE2EDuration="2.331178681s" totalImagesPullingTime="470.472499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:04.330356293 +0000 UTC m=+2232.315476182" watchObservedRunningTime="2026-10-30 03:40:04.331178681 +0000 UTC m=+2232.316298569" Oct 30 03:40:10.301735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:10.301701 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 03:40:11.750340 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:11.750304 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:40:15.573028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:15.572959 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" podUID="369ace70-48d6-4106-b37e-21d5d7535237" containerName="main" containerID="cri-o://39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73" gracePeriod=45 Oct 30 03:40:15.841931 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:15.841906 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:40:16.029311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029277 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\" (UniqueName: \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029449 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\" (UniqueName: \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029497 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029559 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029630 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029765 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029677 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\") pod \"369ace70-48d6-4106-b37e-21d5d7535237\" (UID: \"369ace70-48d6-4106-b37e-21d5d7535237\") " Oct 30 03:40:16.029958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029929 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:16.030049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029947 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:16.030049 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.029996 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:16.030161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.030147 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.030205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.030165 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.030205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.030174 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.031989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.031961 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:40:16.032140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.032122 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:16.032204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.032151 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p" pod "369ace70-48d6-4106-b37e-21d5d7535237" (UID: "369ace70-48d6-4106-b37e-21d5d7535237"). InnerVolumeSpecName "kube-api-access-r8l9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:16.131684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.131604 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8l9p\" (UniqueName: \"kubernetes.io/projected/369ace70-48d6-4106-b37e-21d5d7535237-kube-api-access-r8l9p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.131684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.131630 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369ace70-48d6-4106-b37e-21d5d7535237-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.131684 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.131640 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369ace70-48d6-4106-b37e-21d5d7535237-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:16.359453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.359401 4560 generic.go:356] "Generic (PLEG): container finished" podID="369ace70-48d6-4106-b37e-21d5d7535237" containerID="39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73" exitCode=0 Oct 30 03:40:16.359671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.359461 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" Oct 30 03:40:16.359671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.359470 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" event={"ID":"369ace70-48d6-4106-b37e-21d5d7535237","Type":"ContainerDied","Data":"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73"} Oct 30 03:40:16.359671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.359547 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw" event={"ID":"369ace70-48d6-4106-b37e-21d5d7535237","Type":"ContainerDied","Data":"bd616f4b2a8bbf7e48f0e5b9c25e533bb18735ea1c86d48974547a74773cd946"} Oct 30 03:40:16.359671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.359575 4560 scope.go:122] "RemoveContainer" containerID="39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73" Oct 30 03:40:16.368927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.368909 4560 scope.go:122] "RemoveContainer" containerID="39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73" Oct 30 03:40:16.369183 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:16.369164 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73\": container with ID starting with 39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73 not found: ID does not exist" containerID="39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73" Oct 30 03:40:16.369229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.369194 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73"} err="failed to get container status \"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73\": rpc error: code = NotFound desc = could not find container \"39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73\": container with ID starting with 39f4676ab6c41d43381614cce988118946a7c456c21f6b94b77db8e36a451c73 not found: ID does not exist" Oct 30 03:40:16.412795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.412718 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:40:16.425400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:16.425369 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-22aeb4/noten-model-22aeb4-kserve-5b677bc8fc-jhdgw"] Oct 30 03:40:18.152568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:18.152512 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369ace70-48d6-4106-b37e-21d5d7535237" path="/var/lib/kubelet/pods/369ace70-48d6-4106-b37e-21d5d7535237/volumes" Oct 30 03:40:18.925273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:18.925236 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:40:23.017444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:23.017408 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:40:25.093423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.093380 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 03:40:25.094545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.094499 4560 cpu_manager.go:414] "containerMap: removing container" podUID="369ace70-48d6-4106-b37e-21d5d7535237" containerName="main" Oct 30 03:40:25.094545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.094541 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369ace70-48d6-4106-b37e-21d5d7535237" containerName="main" Oct 30 03:40:25.094705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.094679 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="369ace70-48d6-4106-b37e-21d5d7535237" containerName="main" Oct 30 03:40:25.098428 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.098403 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.115269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.115225 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 03:40:25.136563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.136497 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.136795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.136774 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m92\" (UniqueName: \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.156099 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.156068 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-35f43f\"" Oct 30 03:40:25.156258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.156140 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-35f43f-dockercfg-4vtgh\"" Oct 30 03:40:25.237585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.237548 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m92\" (UniqueName: \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.237787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.237647 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.238612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.238581 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.258425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.258399 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m92\" (UniqueName: \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\") pod \"payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.415064 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.414978 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:25.719424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:25.719398 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 03:40:25.728551 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:25.722428 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f7a842_978b_4921_a056_267eca8e8cab.slice/crio-53ddc820d5fc4f111be8d96b12238f06c1e5ef706830c395172e88182f5bbebe WatchSource:0}: Error finding container 53ddc820d5fc4f111be8d96b12238f06c1e5ef706830c395172e88182f5bbebe: Status 404 returned error can't find the container with id 53ddc820d5fc4f111be8d96b12238f06c1e5ef706830c395172e88182f5bbebe Oct 30 03:40:26.421725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:26.421687 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" event={"ID":"34f7a842-978b-4921-a056-267eca8e8cab","Type":"ContainerStarted","Data":"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15"} Oct 30 03:40:26.422233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:26.421744 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" event={"ID":"34f7a842-978b-4921-a056-267eca8e8cab","Type":"ContainerStarted","Data":"53ddc820d5fc4f111be8d96b12238f06c1e5ef706830c395172e88182f5bbebe"} Oct 30 03:40:26.422233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:26.421789 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:26.461736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:26.461683 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" podStartSLOduration=1.461668314 podStartE2EDuration="1.461668314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:26.459871555 +0000 UTC m=+2254.444991445" watchObservedRunningTime="2026-10-30 03:40:26.461668314 +0000 UTC m=+2254.446788201" Oct 30 03:40:29.969386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:29.969349 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 03:40:29.973989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:29.973964 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:29.992235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:29.992196 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:40:29.993127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:29.993102 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 03:40:30.024391 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.024363 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e10c0f-dockercfg-mqxdc\"" Oct 30 03:40:30.095207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.095176 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6h9x\" (UniqueName: \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.095406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.095390 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.196582 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.196543 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.196825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.196598 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6h9x\" (UniqueName: \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.197365 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.197339 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.216445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.216409 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6h9x\" (UniqueName: \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\") pod \"payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.287599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.287486 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:30.595777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:30.595749 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 03:40:30.596537 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:30.596494 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f3c979_5311_4510_acef_256e2460002a.slice/crio-4057a8c1ee97253cb2fbedda3dc9582cee1a784baa898a8ccc7bd75ea47d99c6 WatchSource:0}: Error finding container 4057a8c1ee97253cb2fbedda3dc9582cee1a784baa898a8ccc7bd75ea47d99c6: Status 404 returned error can't find the container with id 4057a8c1ee97253cb2fbedda3dc9582cee1a784baa898a8ccc7bd75ea47d99c6 Oct 30 03:40:31.449430 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:31.449389 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" event={"ID":"80f3c979-5311-4510-acef-256e2460002a","Type":"ContainerStarted","Data":"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663"} Oct 30 03:40:31.449927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:31.449469 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" event={"ID":"80f3c979-5311-4510-acef-256e2460002a","Type":"ContainerStarted","Data":"4057a8c1ee97253cb2fbedda3dc9582cee1a784baa898a8ccc7bd75ea47d99c6"} Oct 30 03:40:31.449927 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:31.449608 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:31.488583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:31.488494 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" podStartSLOduration=2.488480883 podStartE2EDuration="2.488480883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:31.488233798 +0000 UTC m=+2259.473353690" watchObservedRunningTime="2026-10-30 03:40:31.488480883 +0000 UTC m=+2259.473600771" Oct 30 03:40:33.065043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:33.062364 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:40:33.065043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:33.062720 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" containerID="cri-o://16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b" gracePeriod=30 Oct 30 03:40:33.952560 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:33.952509 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" podUID="bada471d-0075-4300-a439-e156a6233d1b" containerName="main" containerID="cri-o://137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d" gracePeriod=45 Oct 30 03:40:34.119570 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.119493 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" podUID="bada471d-0075-4300-a439-e156a6233d1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": dial tcp 10.132.0.101:8000: connect: connection refused" Oct 30 03:40:34.247351 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.247323 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:40:34.344722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.344690 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.344722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.344729 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.344986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.344773 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.344986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.344862 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.344986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.344906 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.345141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345001 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:34.345141 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345013 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\" (UniqueName: \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\") pod \"bada471d-0075-4300-a439-e156a6233d1b\" (UID: \"bada471d-0075-4300-a439-e156a6233d1b\") " Oct 30 03:40:34.345235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345137 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:34.345235 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345153 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:34.345506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345485 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.345597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345510 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.345597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.345563 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.347124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.347101 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:40:34.347124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.347115 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:34.347406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.347390 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249" pod "bada471d-0075-4300-a439-e156a6233d1b" (UID: "bada471d-0075-4300-a439-e156a6233d1b"). InnerVolumeSpecName "kube-api-access-k4249". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:34.446955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.446919 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4249\" (UniqueName: \"kubernetes.io/projected/bada471d-0075-4300-a439-e156a6233d1b-kube-api-access-k4249\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.446955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.446952 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bada471d-0075-4300-a439-e156a6233d1b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.446955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.446962 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bada471d-0075-4300-a439-e156a6233d1b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:34.468567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.468442 4560 generic.go:356] "Generic (PLEG): container finished" podID="bada471d-0075-4300-a439-e156a6233d1b" containerID="137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d" exitCode=0 Oct 30 03:40:34.468567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.468482 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" event={"ID":"bada471d-0075-4300-a439-e156a6233d1b","Type":"ContainerDied","Data":"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d"} Oct 30 03:40:34.468773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.468461 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" Oct 30 03:40:34.468773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.468561 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2" event={"ID":"bada471d-0075-4300-a439-e156a6233d1b","Type":"ContainerDied","Data":"7f1526cf7b08924361e53aa8c7cf89679c092f84316ac50e8374cf09ac98f4a3"} Oct 30 03:40:34.468773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.468575 4560 scope.go:122] "RemoveContainer" containerID="137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d" Oct 30 03:40:34.481642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.481617 4560 scope.go:122] "RemoveContainer" containerID="137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d" Oct 30 03:40:34.481937 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:34.481917 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d\": container with ID starting with 137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d not found: ID does not exist" containerID="137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d" Oct 30 03:40:34.482032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.481945 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d"} err="failed to get container status \"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d\": rpc error: code = NotFound desc = could not find container \"137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d\": container with ID starting with 137e216a17acc9133c2f6a5cad4f7425dd705893c1ffbc8ec6311bcd496ec37d not found: ID does not exist" Oct 30 03:40:34.520753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.520693 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:40:34.524014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:34.523985 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/auth-test-model-7d99a4-kserve-5ffcc67c45-m7kx2"] Oct 30 03:40:36.155801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:36.155757 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bada471d-0075-4300-a439-e156a6233d1b" path="/var/lib/kubelet/pods/bada471d-0075-4300-a439-e156a6233d1b/volumes" Oct 30 03:40:37.397878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:37.397834 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:40:37.398258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:37.397914 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:40:37.429897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:37.429856 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 03:40:38.039774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.039704 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" containerName="main" containerID="cri-o://e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3" gracePeriod=45 Oct 30 03:40:38.235144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.235053 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": dial tcp 10.132.0.102:8000: connect: connection refused" Oct 30 03:40:38.343599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.343574 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:40:38.376838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.376807 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:40:38.493162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.493127 4560 generic.go:356] "Generic (PLEG): container finished" podID="656b80a9-24a7-4a50-ac32-df076598b41d" containerID="e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3" exitCode=0 Oct 30 03:40:38.493641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.493176 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" Oct 30 03:40:38.493641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.493209 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" event={"ID":"656b80a9-24a7-4a50-ac32-df076598b41d","Type":"ContainerDied","Data":"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3"} Oct 30 03:40:38.493641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.493250 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d" event={"ID":"656b80a9-24a7-4a50-ac32-df076598b41d","Type":"ContainerDied","Data":"8c4e93cadfec5510dc00cf492bd5749a893f6e894e43e812454923ad54904456"} Oct 30 03:40:38.493641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.493268 4560 scope.go:122] "RemoveContainer" containerID="e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3" Oct 30 03:40:38.494640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.494618 4560 generic.go:356] "Generic (PLEG): container finished" podID="d3919fa0-4578-413f-a771-20facbfb73ba" containerID="16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b" exitCode=0 Oct 30 03:40:38.494766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.494665 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" Oct 30 03:40:38.494766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.494696 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" event={"ID":"d3919fa0-4578-413f-a771-20facbfb73ba","Type":"ContainerDied","Data":"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b"} Oct 30 03:40:38.494766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.494746 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl" event={"ID":"d3919fa0-4578-413f-a771-20facbfb73ba","Type":"ContainerDied","Data":"4257f5c678a1e76e3418d1839967e826fcc9fa4e5261178d8317a62db6f063d6"} Oct 30 03:40:38.502252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.502235 4560 scope.go:122] "RemoveContainer" containerID="e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3" Oct 30 03:40:38.502508 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:38.502488 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3\": container with ID starting with e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3 not found: ID does not exist" containerID="e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3" Oct 30 03:40:38.502584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.502529 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3"} err="failed to get container status \"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3\": rpc error: code = NotFound desc = could not find container \"e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3\": container with ID starting with e8871c42396037789813bdd1dc8cf7481949851281f0ff7117feb870372feaf3 not found: ID does not exist" Oct 30 03:40:38.502584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.502549 4560 scope.go:122] "RemoveContainer" containerID="16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b" Oct 30 03:40:38.507735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507714 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.507832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507761 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.507832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507806 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.507952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.507952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507895 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.507952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507933 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.508109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.507977 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.508109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508012 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\" (UniqueName: \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.508109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508050 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.508109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508079 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.508307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508112 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.508307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508191 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\" (UniqueName: \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.508307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508123 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:40:38.508307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508232 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.508307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508284 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\") pod \"d3919fa0-4578-413f-a771-20facbfb73ba\" (UID: \"d3919fa0-4578-413f-a771-20facbfb73ba\") " Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508324 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\") pod \"656b80a9-24a7-4a50-ac32-df076598b41d\" (UID: \"656b80a9-24a7-4a50-ac32-df076598b41d\") " Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508361 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508600 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508765 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508773 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.508809 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509009 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509292 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.509573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509482 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509504 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509552 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509567 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509580 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509595 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509607 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3919fa0-4578-413f-a771-20facbfb73ba-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.509835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.509619 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.510454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.510425 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "kube-api-access-npcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:38.510804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.510777 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "kube-api-access-4kxbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:38.510908 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.510876 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:40:38.511136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.511118 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.511319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.511294 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm" pod "656b80a9-24a7-4a50-ac32-df076598b41d" (UID: "656b80a9-24a7-4a50-ac32-df076598b41d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:38.511383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.511341 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:38.512558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.512537 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo" pod "d3919fa0-4578-413f-a771-20facbfb73ba" (UID: "d3919fa0-4578-413f-a771-20facbfb73ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:40:38.513105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.513091 4560 scope.go:122] "RemoveContainer" containerID="16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b" Oct 30 03:40:38.513386 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:38.513359 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b\": container with ID starting with 16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b not found: ID does not exist" containerID="16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b" Oct 30 03:40:38.513427 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.513394 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b"} err="failed to get container status \"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b\": rpc error: code = NotFound desc = could not find container \"16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b\": container with ID starting with 16cf6df6ce46f14c2ad04fb15883408b12584cd96f9199223647afd9e128522b not found: ID does not exist" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610651 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3919fa0-4578-413f-a771-20facbfb73ba-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610679 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npcjc\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-kube-api-access-npcjc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610690 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3919fa0-4578-413f-a771-20facbfb73ba-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610698 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/656b80a9-24a7-4a50-ac32-df076598b41d-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610707 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxbf\" (UniqueName: \"kubernetes.io/projected/656b80a9-24a7-4a50-ac32-df076598b41d-kube-api-access-4kxbf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610718 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3919fa0-4578-413f-a771-20facbfb73ba-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.610729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.610731 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/656b80a9-24a7-4a50-ac32-df076598b41d-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:40:38.844543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.844501 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:40:38.847397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.847372 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-wkqvl"] Oct 30 03:40:38.890800 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.890722 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:40:38.894667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:38.894642 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/auth-test-model-852107-kserve-7fc4c79ffc-pbl8d"] Oct 30 03:40:39.346149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346119 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 03:40:39.346686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346669 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" Oct 30 03:40:39.346749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346688 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" Oct 30 03:40:39.346749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346705 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bada471d-0075-4300-a439-e156a6233d1b" containerName="main" Oct 30 03:40:39.346749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346711 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bada471d-0075-4300-a439-e156a6233d1b" containerName="main" Oct 30 03:40:39.346749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346723 4560 cpu_manager.go:414] "containerMap: removing container" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" containerName="main" Oct 30 03:40:39.346749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346729 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" containerName="main" Oct 30 03:40:39.346904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346817 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bada471d-0075-4300-a439-e156a6233d1b" containerName="main" Oct 30 03:40:39.346904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346827 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" containerName="main" Oct 30 03:40:39.346904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.346837 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" containerName="istio-proxy" Oct 30 03:40:39.351612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.351585 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.367998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.367967 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-93b821\"" Oct 30 03:40:39.368207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.368123 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-93b821-dockercfg-m82nf\"" Oct 30 03:40:39.368347 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.368330 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 03:40:39.474642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.474605 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 03:40:39.478556 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.478530 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.501851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.501812 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 03:40:39.521059 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.521027 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjc9\" (UniqueName: \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.521234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.521194 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.622020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.621937 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjc9\" (UniqueName: \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.622189 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.622029 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.622189 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.622072 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrzj\" (UniqueName: \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.622189 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.622101 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.622816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.622795 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.641461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.641431 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjc9\" (UniqueName: \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\") pod \"payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.666507 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.666475 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:39.723273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.723235 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.723403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.723345 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrzj\" (UniqueName: \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.724030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.724006 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.744028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.742700 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrzj\" (UniqueName: \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\") pod \"payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.795302 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.795268 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:39.968363 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:39.968329 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 03:40:39.969424 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:39.969399 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb0bb28_a713_4d4b_b5c4_b882416ddea7.slice/crio-9a161f2fc58846b062d56f8559d5c7dacccc6aa9bf9d89d413d90e57e44e0005 WatchSource:0}: Error finding container 9a161f2fc58846b062d56f8559d5c7dacccc6aa9bf9d89d413d90e57e44e0005: Status 404 returned error can't find the container with id 9a161f2fc58846b062d56f8559d5c7dacccc6aa9bf9d89d413d90e57e44e0005 Oct 30 03:40:40.098606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.098577 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 03:40:40.099469 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:40.099434 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47f8638_835e_422b_bb51_2dc58acba6e1.slice/crio-37b6532ecd0f8e95cdf18e90b1b60d08e808c0db5df7de1bb123444514c015cd WatchSource:0}: Error finding container 37b6532ecd0f8e95cdf18e90b1b60d08e808c0db5df7de1bb123444514c015cd: Status 404 returned error can't find the container with id 37b6532ecd0f8e95cdf18e90b1b60d08e808c0db5df7de1bb123444514c015cd Oct 30 03:40:40.155273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.155239 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656b80a9-24a7-4a50-ac32-df076598b41d" path="/var/lib/kubelet/pods/656b80a9-24a7-4a50-ac32-df076598b41d/volumes" Oct 30 03:40:40.155839 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.155819 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3919fa0-4578-413f-a771-20facbfb73ba" path="/var/lib/kubelet/pods/d3919fa0-4578-413f-a771-20facbfb73ba/volumes" Oct 30 03:40:40.510224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.510174 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" event={"ID":"1eb0bb28-a713-4d4b-b5c4-b882416ddea7","Type":"ContainerStarted","Data":"6c2561d24d74ddf0a2d9bd2b27d5256c65702889b5ab2c4f8a52a54cb3eeb3dc"} Oct 30 03:40:40.510689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.510239 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" event={"ID":"1eb0bb28-a713-4d4b-b5c4-b882416ddea7","Type":"ContainerStarted","Data":"9a161f2fc58846b062d56f8559d5c7dacccc6aa9bf9d89d413d90e57e44e0005"} Oct 30 03:40:40.510689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.510268 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:40.511797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.511776 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" event={"ID":"a47f8638-835e-422b-bb51-2dc58acba6e1","Type":"ContainerStarted","Data":"cd4e538d322c4208105fb3fb5e5a7a8601f1a3aba5f90093e91b4cd5ddd3bc39"} Oct 30 03:40:40.511880 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.511814 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" event={"ID":"a47f8638-835e-422b-bb51-2dc58acba6e1","Type":"ContainerStarted","Data":"37b6532ecd0f8e95cdf18e90b1b60d08e808c0db5df7de1bb123444514c015cd"} Oct 30 03:40:40.511922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.511901 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:40.551014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.550961 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" podStartSLOduration=1.550947606 podStartE2EDuration="1.550947606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:40.548261649 +0000 UTC m=+2268.533381537" watchObservedRunningTime="2026-10-30 03:40:40.550947606 +0000 UTC m=+2268.536067494" Oct 30 03:40:40.588005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:40.587954 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" podStartSLOduration=1.587940522 podStartE2EDuration="1.587940522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:40.586905688 +0000 UTC m=+2268.572025576" watchObservedRunningTime="2026-10-30 03:40:40.587940522 +0000 UTC m=+2268.573060411" Oct 30 03:40:41.750160 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:40:41.750126 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:40:42.458218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.458189 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 03:40:42.939096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.939008 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:40:42.943430 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.943407 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:42.960788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.960735 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dbed6d\"" Oct 30 03:40:42.960933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.960913 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:40:42.961628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.961591 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:42.961722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.961660 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g48p\" (UniqueName: \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:42.965420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:42.965397 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dbed6d-dockercfg-dwmbb\"" Oct 30 03:40:43.038740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.038706 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:40:43.043176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.043148 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.062144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.062112 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:40:43.062638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.062614 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.062813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.062658 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjcn\" (UniqueName: \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.062813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.062692 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:43.062917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.062832 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g48p\" (UniqueName: \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:43.063611 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.063587 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:43.082015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.081987 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g48p\" (UniqueName: \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\") pod \"payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:43.163916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.163883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.164107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.163934 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjcn\" (UniqueName: \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.164655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.164635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.183728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.183696 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjcn\" (UniqueName: \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\") pod \"payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.257042 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.257010 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:43.360661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.360619 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:43.562918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.562860 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:40:43.668954 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:43.668919 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413f155b_973f_4915_9e25_31197d5f7823.slice/crio-f9e86e4e35ceda7b2477837dc9585fa0803f205d0faba03d5eeafecc6011f65d WatchSource:0}: Error finding container f9e86e4e35ceda7b2477837dc9585fa0803f205d0faba03d5eeafecc6011f65d: Status 404 returned error can't find the container with id f9e86e4e35ceda7b2477837dc9585fa0803f205d0faba03d5eeafecc6011f65d Oct 30 03:40:43.669403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:43.669383 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:40:44.536073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.536037 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" event={"ID":"a38d96fc-954c-4c45-a16a-3b73082537d1","Type":"ContainerStarted","Data":"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75"} Oct 30 03:40:44.536613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.536101 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" event={"ID":"a38d96fc-954c-4c45-a16a-3b73082537d1","Type":"ContainerStarted","Data":"8a6d05d4f155eb4a9a73fd8e5913f96f9529cb995ffc98026b9315890006821d"} Oct 30 03:40:44.536613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.536293 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:44.537815 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.537790 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" event={"ID":"413f155b-973f-4915-9e25-31197d5f7823","Type":"ContainerStarted","Data":"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e"} Oct 30 03:40:44.537921 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.537834 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" event={"ID":"413f155b-973f-4915-9e25-31197d5f7823","Type":"ContainerStarted","Data":"f9e86e4e35ceda7b2477837dc9585fa0803f205d0faba03d5eeafecc6011f65d"} Oct 30 03:40:44.537921 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.537867 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:44.581176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.581116 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" podStartSLOduration=2.581098812 podStartE2EDuration="2.581098812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:44.578422441 +0000 UTC m=+2272.563542340" watchObservedRunningTime="2026-10-30 03:40:44.581098812 +0000 UTC m=+2272.566218702" Oct 30 03:40:44.620018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:44.619968 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" podStartSLOduration=1.619953143 podStartE2EDuration="1.619953143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:44.618009792 +0000 UTC m=+2272.603129681" watchObservedRunningTime="2026-10-30 03:40:44.619953143 +0000 UTC m=+2272.605073065" Oct 30 03:40:47.280626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.280590 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 03:40:47.285162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.285136 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.318481 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.318444 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 03:40:47.318682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.318562 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-35f43f-serving-cert\"" Oct 30 03:40:47.322606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.322570 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.322740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.322628 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.322740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.322673 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.322740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.322720 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.423763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.423719 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.423949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.423866 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.423949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.423896 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.423949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.423922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.424654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.424626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.426575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.426545 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.427172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.427149 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.444802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.444764 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\") pod \"maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.601293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.601219 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:47.914231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.914205 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 03:40:47.914691 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:47.914668 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fc4d2_134e_44b9_aa65_7385b9f96b19.slice/crio-009c2c49ae32eb8bc55c61761de8e947c97e92240d657295d5ca8057153f443c WatchSource:0}: Error finding container 009c2c49ae32eb8bc55c61761de8e947c97e92240d657295d5ca8057153f443c: Status 404 returned error can't find the container with id 009c2c49ae32eb8bc55c61761de8e947c97e92240d657295d5ca8057153f443c Oct 30 03:40:47.916139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:47.916118 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:40:48.569900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:48.569786 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" event={"ID":"087fc4d2-134e-44b9-aa65-7385b9f96b19","Type":"ContainerStarted","Data":"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac"} Oct 30 03:40:48.569900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:48.569871 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" event={"ID":"087fc4d2-134e-44b9-aa65-7385b9f96b19","Type":"ContainerStarted","Data":"009c2c49ae32eb8bc55c61761de8e947c97e92240d657295d5ca8057153f443c"} Oct 30 03:40:48.570447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:48.570125 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:48.614042 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:48.613989 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" podStartSLOduration=1.258458652 podStartE2EDuration="1.613974323s" totalImagesPullingTime="355.515671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:48.611651216 +0000 UTC m=+2276.596771105" watchObservedRunningTime="2026-10-30 03:40:48.613974323 +0000 UTC m=+2276.599094221" Oct 30 03:40:51.520314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:51.520278 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 03:40:51.520916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:51.520358 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 03:40:52.620891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.620849 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 03:40:52.625061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.625040 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.643306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.643280 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 03:40:52.685130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.685094 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.685311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.685152 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-398c25\"" Oct 30 03:40:52.685311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.685153 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.685311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.685297 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-398c25-dockercfg-zc4nr\"" Oct 30 03:40:52.786080 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.786035 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.786253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.786096 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.786858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.786839 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.805538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.805486 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\") pod \"payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:52.938333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:52.938304 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:53.241961 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:53.241927 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 03:40:53.245148 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:53.245121 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c721f6_07db_4192_9b56_cdb5b85cde3f.slice/crio-415e1f8c213d406cd3023e3f4975f7e89909379500e6ef303e860f63a053b5ae WatchSource:0}: Error finding container 415e1f8c213d406cd3023e3f4975f7e89909379500e6ef303e860f63a053b5ae: Status 404 returned error can't find the container with id 415e1f8c213d406cd3023e3f4975f7e89909379500e6ef303e860f63a053b5ae Oct 30 03:40:53.601583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:53.601490 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" event={"ID":"02c721f6-07db-4192-9b56-cdb5b85cde3f","Type":"ContainerStarted","Data":"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113"} Oct 30 03:40:53.601583 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:53.601565 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" event={"ID":"02c721f6-07db-4192-9b56-cdb5b85cde3f","Type":"ContainerStarted","Data":"415e1f8c213d406cd3023e3f4975f7e89909379500e6ef303e860f63a053b5ae"} Oct 30 03:40:53.601775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:53.601712 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:40:53.642004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:53.641952 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" podStartSLOduration=1.6419371969999998 podStartE2EDuration="1.641937197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:53.639713268 +0000 UTC m=+2281.624833147" watchObservedRunningTime="2026-10-30 03:40:53.641937197 +0000 UTC m=+2281.627057087" Oct 30 03:40:54.580780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:54.580748 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 03:40:55.238055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.238021 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 03:40:55.242289 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.242272 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.270587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.270560 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e10c0f-serving-cert\"" Oct 30 03:40:55.274225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.274199 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 03:40:55.315573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.315541 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.315794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.315767 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.315850 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.315812 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.315949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.315921 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr2kh\" (UniqueName: \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.417619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.417582 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.417619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.417621 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.417857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.417753 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2kh\" (UniqueName: \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.417857 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.417820 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.418356 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.418331 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.420120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.420099 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.420266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.420248 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.436956 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.436931 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2kh\" (UniqueName: \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\") pod \"maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.546948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.546871 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:40:55.546948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.546933 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:40:55.555652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.555633 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:55.874987 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:55.874955 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 03:40:55.876483 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:40:55.876455 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69251e1c_69e6_49ed_ac90_6d8edb9f64ca.slice/crio-778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69 WatchSource:0}: Error finding container 778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69: Status 404 returned error can't find the container with id 778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69 Oct 30 03:40:56.624167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:56.624082 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" event={"ID":"69251e1c-69e6-49ed-ac90-6d8edb9f64ca","Type":"ContainerStarted","Data":"a7cc62ac43e12846f507b8519a6ba17a55290285d9ea8d269d0d6fb22721299d"} Oct 30 03:40:56.624167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:56.624134 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" event={"ID":"69251e1c-69e6-49ed-ac90-6d8edb9f64ca","Type":"ContainerStarted","Data":"778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69"} Oct 30 03:40:56.624679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:56.624183 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:40:56.666574 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:40:56.666502 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" podStartSLOduration=1.208117354 podStartE2EDuration="1.666488361s" totalImagesPullingTime="458.371007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:56.663951889 +0000 UTC m=+2284.649071779" watchObservedRunningTime="2026-10-30 03:40:56.666488361 +0000 UTC m=+2284.651608287" Oct 30 03:41:02.636316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:02.636286 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 03:41:04.610476 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:04.610403 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 03:41:05.689209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.689177 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 03:41:05.694779 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.694757 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.715101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.715073 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 03:41:05.723698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.723664 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-93b821-serving-cert\"" Oct 30 03:41:05.836901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.836866 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.837079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.837009 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zpc\" (UniqueName: \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.837079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.837063 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.837166 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.837150 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.938014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.937980 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.938172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.938066 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.938172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.938123 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zpc\" (UniqueName: \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.938172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.938165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.938765 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.938741 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.940628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.940564 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.940628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.940577 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:05.957471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:05.957442 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zpc\" (UniqueName: \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\") pod \"maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:06.009208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.009174 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:06.315670 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.315645 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 03:41:06.316875 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:41:06.316844 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9856027_c513_4c98_b2af_f8834eede869.slice/crio-e7dd18a38d464ebf401172be1cf2f8bd0251e8137ef428cc02e5d3f5c33272ee WatchSource:0}: Error finding container e7dd18a38d464ebf401172be1cf2f8bd0251e8137ef428cc02e5d3f5c33272ee: Status 404 returned error can't find the container with id e7dd18a38d464ebf401172be1cf2f8bd0251e8137ef428cc02e5d3f5c33272ee Oct 30 03:41:06.684286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.684254 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" event={"ID":"e9856027-c513-4c98-b2af-f8834eede869","Type":"ContainerStarted","Data":"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e"} Oct 30 03:41:06.684446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.684311 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" event={"ID":"e9856027-c513-4c98-b2af-f8834eede869","Type":"ContainerStarted","Data":"e7dd18a38d464ebf401172be1cf2f8bd0251e8137ef428cc02e5d3f5c33272ee"} Oct 30 03:41:06.684446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.684340 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:06.726186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:06.726131 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" podStartSLOduration=1.43938974 podStartE2EDuration="1.726115873s" totalImagesPullingTime="286.726133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:06.724297186 +0000 UTC m=+2294.709417090" watchObservedRunningTime="2026-10-30 03:41:06.726115873 +0000 UTC m=+2294.711235760" Oct 30 03:41:11.749043 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:41:11.749009 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:41:12.696142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:12.696115 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 03:41:20.799659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.799625 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 03:41:20.803417 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.803400 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:20.822359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.822274 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-398c25-serving-cert\"" Oct 30 03:41:20.823245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.823219 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 03:41:20.911279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.911238 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:20.911279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.911276 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4mq\" (UniqueName: \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:20.911559 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.911296 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:20.911559 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:20.911463 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.012345 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.012310 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.012534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.012353 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4mq\" (UniqueName: \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.012534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.012381 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.012534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.012452 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.013133 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.013106 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.014911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.014887 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.015013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.014944 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.031447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.031420 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4mq\" (UniqueName: \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\") pod \"maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.117404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.117319 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:21.425693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.425660 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 03:41:21.427173 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:41:21.427147 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e2d3b3_174f_4524_a2f1_934813a0655a.slice/crio-2407939a8e63cf87aab2223c8a1c63b8336b286d121f5873f08a07794a91f804 WatchSource:0}: Error finding container 2407939a8e63cf87aab2223c8a1c63b8336b286d121f5873f08a07794a91f804: Status 404 returned error can't find the container with id 2407939a8e63cf87aab2223c8a1c63b8336b286d121f5873f08a07794a91f804 Oct 30 03:41:21.770963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:21.770926 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" event={"ID":"68e2d3b3-174f-4524-a2f1-934813a0655a","Type":"ContainerStarted","Data":"2407939a8e63cf87aab2223c8a1c63b8336b286d121f5873f08a07794a91f804"} Oct 30 03:41:22.781632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:22.781593 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" event={"ID":"68e2d3b3-174f-4524-a2f1-934813a0655a","Type":"ContainerStarted","Data":"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af"} Oct 30 03:41:22.782109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:22.781659 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:22.821798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:22.821738 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" podStartSLOduration=2.41295817 podStartE2EDuration="2.82172038s" totalImagesPullingTime="408.76221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:22.82127613 +0000 UTC m=+2310.806396018" watchObservedRunningTime="2026-10-30 03:41:22.82172038 +0000 UTC m=+2310.806840269" Oct 30 03:41:28.794381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:28.794351 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 03:41:41.749428 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:41:41.749389 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:41:48.877459 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.877424 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:41:48.881665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.881637 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.903588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.903556 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:41:48.913403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.913374 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dbed6d\"/\"autores-model-dbed6d-kserve-self-signed-certs\"" Oct 30 03:41:48.913625 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.913446 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dbed6d\"/\"default-dockercfg-khqxg\"" Oct 30 03:41:48.945646 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.945433 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7864f\" (UniqueName: \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.945820 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.945726 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.945820 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.945774 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.945917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.945898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.946013 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.945997 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.946079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.946063 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:48.962165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.962144 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dbed6d\"/\"kube-root-ca.crt\"" Oct 30 03:41:48.962297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:48.962236 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dbed6d\"/\"openshift-service-ca.crt\"" Oct 30 03:41:49.047287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047248 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047303 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047350 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047410 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047461 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047501 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7864f\" (UniqueName: \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047908 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047882 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.047970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.047941 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.048196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.048171 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.049863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.049834 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.050028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.050009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.070327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.070287 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7864f\" (UniqueName: \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\") pod \"autores-model-dbed6d-kserve-66b56d75c-cdtk6\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.201114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.201025 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:49.510310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.510280 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:41:49.511566 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:41:49.511536 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f23d002_e2ea_4610_9260_c51c5ba8df02.slice/crio-2cf80f4765ee92a4e39d3c37a79bdd66a1883336180f50d3e79948a4a31192ee WatchSource:0}: Error finding container 2cf80f4765ee92a4e39d3c37a79bdd66a1883336180f50d3e79948a4a31192ee: Status 404 returned error can't find the container with id 2cf80f4765ee92a4e39d3c37a79bdd66a1883336180f50d3e79948a4a31192ee Oct 30 03:41:49.940127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.940035 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" event={"ID":"9f23d002-e2ea-4610-9260-c51c5ba8df02","Type":"ContainerStarted","Data":"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e"} Oct 30 03:41:49.940127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.940077 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" event={"ID":"9f23d002-e2ea-4610-9260-c51c5ba8df02","Type":"ContainerStarted","Data":"2cf80f4765ee92a4e39d3c37a79bdd66a1883336180f50d3e79948a4a31192ee"} Oct 30 03:41:49.981819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:49.981765 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" podStartSLOduration=1.981750173 podStartE2EDuration="1.981750173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:49.978655231 +0000 UTC m=+2337.963775120" watchObservedRunningTime="2026-10-30 03:41:49.981750173 +0000 UTC m=+2337.966870098" Oct 30 03:41:50.201569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:50.201544 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:59.201452 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:59.201408 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:41:59.214703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:41:59.214670 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:42:00.024917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:00.024883 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:42:04.775076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.775041 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 03:42:04.780739 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.780719 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.797581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.797549 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 03:42:04.807100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.807074 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-855e14-dockercfg-wsmct\"" Oct 30 03:42:04.828550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.828498 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-855e14\"" Oct 30 03:42:04.837055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.837012 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rw9r\" (UniqueName: \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.837239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.837154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.938625 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.938587 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rw9r\" (UniqueName: \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.938801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.938665 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.939419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.939398 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:04.958120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:04.958090 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rw9r\" (UniqueName: \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\") pod \"payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:05.095865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:05.095783 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:05.398091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:05.398058 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 03:42:05.399492 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:42:05.399449 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceba5f9_ae00_4ea7_9fff_216955705e84.slice/crio-fd71d460761732d6bcc7fdd422e1f95daf945f2038dc75361283b2d348a461b6 WatchSource:0}: Error finding container fd71d460761732d6bcc7fdd422e1f95daf945f2038dc75361283b2d348a461b6: Status 404 returned error can't find the container with id fd71d460761732d6bcc7fdd422e1f95daf945f2038dc75361283b2d348a461b6 Oct 30 03:42:06.050673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:06.050632 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" event={"ID":"fceba5f9-ae00-4ea7-9fff-216955705e84","Type":"ContainerStarted","Data":"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767"} Oct 30 03:42:06.051155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:06.050695 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" event={"ID":"fceba5f9-ae00-4ea7-9fff-216955705e84","Type":"ContainerStarted","Data":"fd71d460761732d6bcc7fdd422e1f95daf945f2038dc75361283b2d348a461b6"} Oct 30 03:42:06.051155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:06.050786 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:06.092733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:06.092677 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" podStartSLOduration=2.092664582 podStartE2EDuration="2.092664582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:06.089562989 +0000 UTC m=+2354.074682878" watchObservedRunningTime="2026-10-30 03:42:06.092664582 +0000 UTC m=+2354.077784469" Oct 30 03:42:11.749350 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:42:11.749312 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:42:15.361727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:15.361688 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:42:17.060440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:17.060411 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 03:42:30.383538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.383467 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" podUID="9f23d002-e2ea-4610-9260-c51c5ba8df02" containerName="main" containerID="cri-o://cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e" gracePeriod=45 Oct 30 03:42:30.655909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.655838 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:42:30.728813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.728781 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.728858 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\" (UniqueName: \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.728922 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.728961 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729028 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729174 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729060 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\") pod \"9f23d002-e2ea-4610-9260-c51c5ba8df02\" (UID: \"9f23d002-e2ea-4610-9260-c51c5ba8df02\") " Oct 30 03:42:30.729384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729357 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:42:30.729504 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729484 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:42:30.729722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729699 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:30.729819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729728 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:30.729884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.729832 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:42:30.731509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.731481 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:42:30.731911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.731891 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:42:30.732057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.732041 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f" pod "9f23d002-e2ea-4610-9260-c51c5ba8df02" (UID: "9f23d002-e2ea-4610-9260-c51c5ba8df02"). InnerVolumeSpecName "kube-api-access-7864f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:42:30.831244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.831209 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:30.831244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.831239 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f23d002-e2ea-4610-9260-c51c5ba8df02-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:30.831244 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.831248 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7864f\" (UniqueName: \"kubernetes.io/projected/9f23d002-e2ea-4610-9260-c51c5ba8df02-kube-api-access-7864f\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:30.831484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:30.831260 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f23d002-e2ea-4610-9260-c51c5ba8df02-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:42:31.207905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.207865 4560 generic.go:356] "Generic (PLEG): container finished" podID="9f23d002-e2ea-4610-9260-c51c5ba8df02" containerID="cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e" exitCode=0 Oct 30 03:42:31.208104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.207908 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" Oct 30 03:42:31.208104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.207955 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" event={"ID":"9f23d002-e2ea-4610-9260-c51c5ba8df02","Type":"ContainerDied","Data":"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e"} Oct 30 03:42:31.208104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.208005 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6" event={"ID":"9f23d002-e2ea-4610-9260-c51c5ba8df02","Type":"ContainerDied","Data":"2cf80f4765ee92a4e39d3c37a79bdd66a1883336180f50d3e79948a4a31192ee"} Oct 30 03:42:31.208104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.208031 4560 scope.go:122] "RemoveContainer" containerID="cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e" Oct 30 03:42:31.218767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.218750 4560 scope.go:122] "RemoveContainer" containerID="cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e" Oct 30 03:42:31.219028 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:42:31.219010 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e\": container with ID starting with cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e not found: ID does not exist" containerID="cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e" Oct 30 03:42:31.219077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.219035 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e"} err="failed to get container status \"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e\": rpc error: code = NotFound desc = could not find container \"cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e\": container with ID starting with cb4e101fe5d53778a3ade9c2841302da6a1ae0a26f3ea357e46f6009984e134e not found: ID does not exist" Oct 30 03:42:31.260107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.260073 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:42:31.280823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:31.273932 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-dbed6d/autores-model-dbed6d-kserve-66b56d75c-cdtk6"] Oct 30 03:42:32.154193 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:32.154156 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f23d002-e2ea-4610-9260-c51c5ba8df02" path="/var/lib/kubelet/pods/9f23d002-e2ea-4610-9260-c51c5ba8df02/volumes" Oct 30 03:42:34.601728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.601645 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 03:42:34.602475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.602452 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9f23d002-e2ea-4610-9260-c51c5ba8df02" containerName="main" Oct 30 03:42:34.602475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.602475 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f23d002-e2ea-4610-9260-c51c5ba8df02" containerName="main" Oct 30 03:42:34.602675 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.602632 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f23d002-e2ea-4610-9260-c51c5ba8df02" containerName="main" Oct 30 03:42:34.607554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.607530 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.635053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.634993 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-855e14-serving-cert\"" Oct 30 03:42:34.639954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.639922 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 03:42:34.673703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.673665 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.673922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.673793 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.673922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.673842 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kxz\" (UniqueName: \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.674101 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.674059 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.775157 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.775119 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.775359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.775181 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kxz\" (UniqueName: \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.775433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.775362 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.775433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.775398 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.776025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.775999 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.777692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.777666 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.777848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.777827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.794289 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.794266 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kxz\" (UniqueName: \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\") pod \"maas-api-e2e-extmod-b-855e14-578498768-2zsv8\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:34.921333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:34.921242 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:35.233898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:35.231089 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 03:42:36.252620 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:36.252586 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" event={"ID":"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9","Type":"ContainerStarted","Data":"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67"} Oct 30 03:42:36.253096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:36.252643 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" event={"ID":"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9","Type":"ContainerStarted","Data":"44285fa04dcde0a893dba053b200e6f158d7383608a89446c257721bef8b8c35"} Oct 30 03:42:36.253096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:36.252665 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:36.294221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:36.294167 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" podStartSLOduration=1.944590449 podStartE2EDuration="2.294153816s" totalImagesPullingTime="349.563367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:36.292977315 +0000 UTC m=+2384.278097203" watchObservedRunningTime="2026-10-30 03:42:36.294153816 +0000 UTC m=+2384.279273705" Oct 30 03:42:41.751081 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:42:41.751045 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:42:42.264957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:42.264929 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 03:42:58.822453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:58.822411 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:42:58.855501 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:42:58.855465 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:43:03.823185 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:03.823144 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" podUID="413f155b-973f-4915-9e25-31197d5f7823" containerName="payload-processing" containerID="cri-o://e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e" gracePeriod=25 Oct 30 03:43:03.856660 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:03.856621 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" podUID="a38d96fc-954c-4c45-a16a-3b73082537d1" containerName="payload-pre-processing" containerID="cri-o://54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75" gracePeriod=25 Oct 30 03:43:04.128617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.128592 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:43:04.155808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.155785 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:43:04.202568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.202531 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\") pod \"413f155b-973f-4915-9e25-31197d5f7823\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " Oct 30 03:43:04.202754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.202670 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\" (UniqueName: \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\") pod \"413f155b-973f-4915-9e25-31197d5f7823\" (UID: \"413f155b-973f-4915-9e25-31197d5f7823\") " Oct 30 03:43:04.202820 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.202776 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\") pod \"a38d96fc-954c-4c45-a16a-3b73082537d1\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " Oct 30 03:43:04.202891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.202827 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\" (UniqueName: \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\") pod \"a38d96fc-954c-4c45-a16a-3b73082537d1\" (UID: \"a38d96fc-954c-4c45-a16a-3b73082537d1\") " Oct 30 03:43:04.202947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.202913 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume" pod "413f155b-973f-4915-9e25-31197d5f7823" (UID: "413f155b-973f-4915-9e25-31197d5f7823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:04.203277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.203249 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume" pod "a38d96fc-954c-4c45-a16a-3b73082537d1" (UID: "a38d96fc-954c-4c45-a16a-3b73082537d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:04.203435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.203411 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d96fc-954c-4c45-a16a-3b73082537d1-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:43:04.203435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.203434 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413f155b-973f-4915-9e25-31197d5f7823-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:43:04.205779 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.205502 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p" pod "a38d96fc-954c-4c45-a16a-3b73082537d1" (UID: "a38d96fc-954c-4c45-a16a-3b73082537d1"). InnerVolumeSpecName "kube-api-access-4g48p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:04.205779 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.205723 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn" pod "413f155b-973f-4915-9e25-31197d5f7823" (UID: "413f155b-973f-4915-9e25-31197d5f7823"). InnerVolumeSpecName "kube-api-access-cbjcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:04.304741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.304707 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g48p\" (UniqueName: \"kubernetes.io/projected/a38d96fc-954c-4c45-a16a-3b73082537d1-kube-api-access-4g48p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:43:04.304741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.304735 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjcn\" (UniqueName: \"kubernetes.io/projected/413f155b-973f-4915-9e25-31197d5f7823-kube-api-access-cbjcn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:43:04.427678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.427589 4560 generic.go:356] "Generic (PLEG): container finished" podID="413f155b-973f-4915-9e25-31197d5f7823" containerID="e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e" exitCode=0 Oct 30 03:43:04.427678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.427637 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" Oct 30 03:43:04.427887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.427676 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" event={"ID":"413f155b-973f-4915-9e25-31197d5f7823","Type":"ContainerDied","Data":"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e"} Oct 30 03:43:04.427887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.427738 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44" event={"ID":"413f155b-973f-4915-9e25-31197d5f7823","Type":"ContainerDied","Data":"f9e86e4e35ceda7b2477837dc9585fa0803f205d0faba03d5eeafecc6011f65d"} Oct 30 03:43:04.427887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.427769 4560 scope.go:122] "RemoveContainer" containerID="e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e" Oct 30 03:43:04.429044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.429023 4560 generic.go:356] "Generic (PLEG): container finished" podID="a38d96fc-954c-4c45-a16a-3b73082537d1" containerID="54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75" exitCode=0 Oct 30 03:43:04.429159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.429050 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" event={"ID":"a38d96fc-954c-4c45-a16a-3b73082537d1","Type":"ContainerDied","Data":"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75"} Oct 30 03:43:04.429159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.429061 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" Oct 30 03:43:04.429159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.429077 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9" event={"ID":"a38d96fc-954c-4c45-a16a-3b73082537d1","Type":"ContainerDied","Data":"8a6d05d4f155eb4a9a73fd8e5913f96f9529cb995ffc98026b9315890006821d"} Oct 30 03:43:04.438594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.438568 4560 scope.go:122] "RemoveContainer" containerID="e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e" Oct 30 03:43:04.438901 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:43:04.438881 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e\": container with ID starting with e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e not found: ID does not exist" containerID="e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e" Oct 30 03:43:04.438998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.438909 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e"} err="failed to get container status \"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e\": rpc error: code = NotFound desc = could not find container \"e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e\": container with ID starting with e6c0c2dd44b00482afe284ae5ebcf531cdde6f55f3318f14ead5dcbe4849753e not found: ID does not exist" Oct 30 03:43:04.438998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.438925 4560 scope.go:122] "RemoveContainer" containerID="54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75" Oct 30 03:43:04.447687 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.447670 4560 scope.go:122] "RemoveContainer" containerID="54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75" Oct 30 03:43:04.447949 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:43:04.447928 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75\": container with ID starting with 54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75 not found: ID does not exist" containerID="54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75" Oct 30 03:43:04.448005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.447957 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75"} err="failed to get container status \"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75\": rpc error: code = NotFound desc = could not find container \"54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75\": container with ID starting with 54b2901ccdcea815646dd9933bae20b6e308877d97492c5f820da890b27c7f75 not found: ID does not exist" Oct 30 03:43:04.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.479256 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:43:04.482901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.482877 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dbed6d-5d9898d5f-g7jx9"] Oct 30 03:43:04.515662 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.515631 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:43:04.521064 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:04.521042 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dbed6d-54984d74c8-4cb44"] Oct 30 03:43:06.152897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:06.152860 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413f155b-973f-4915-9e25-31197d5f7823" path="/var/lib/kubelet/pods/413f155b-973f-4915-9e25-31197d5f7823/volumes" Oct 30 03:43:06.153472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:06.153451 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38d96fc-954c-4c45-a16a-3b73082537d1" path="/var/lib/kubelet/pods/a38d96fc-954c-4c45-a16a-3b73082537d1/volumes" Oct 30 03:43:11.756777 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:43:11.756732 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:43:14.078245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.078202 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 03:43:14.079016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.078992 4560 cpu_manager.go:414] "containerMap: removing container" podUID="413f155b-973f-4915-9e25-31197d5f7823" containerName="payload-processing" Oct 30 03:43:14.079114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.079017 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413f155b-973f-4915-9e25-31197d5f7823" containerName="payload-processing" Oct 30 03:43:14.079114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.079082 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a38d96fc-954c-4c45-a16a-3b73082537d1" containerName="payload-pre-processing" Oct 30 03:43:14.079114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.079092 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38d96fc-954c-4c45-a16a-3b73082537d1" containerName="payload-pre-processing" Oct 30 03:43:14.079273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.079247 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="413f155b-973f-4915-9e25-31197d5f7823" containerName="payload-processing" Oct 30 03:43:14.079273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.079265 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38d96fc-954c-4c45-a16a-3b73082537d1" containerName="payload-pre-processing" Oct 30 03:43:14.084108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.084079 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.114104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.114059 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 03:43:14.186967 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.186940 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-855e14-openshift-default-dockercfg-24jzw\"" Oct 30 03:43:14.217288 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217259 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217299 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217324 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217702 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217563 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6g2d\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217702 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217664 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217809 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217797 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217853 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.217925 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.217915 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319547 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319579 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319609 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319668 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319692 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.319720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319718 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319769 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319825 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6g2d\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.319969 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.320032 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.320108 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.320289 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.320423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.320403 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.321979 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.321949 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.322418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.322392 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.339008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.338945 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.339224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.339202 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6g2d\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.397740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.397701 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:14.720476 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.720445 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 03:43:14.721106 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:43:14.721076 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7554b9fe_6f88_48a3_bc9d_4891493ad5a9.slice/crio-f45cf3e4a8b3f2c2b30292144fa89006b1f9cb996876a38d35804b61f4834f89 WatchSource:0}: Error finding container f45cf3e4a8b3f2c2b30292144fa89006b1f9cb996876a38d35804b61f4834f89: Status 404 returned error can't find the container with id f45cf3e4a8b3f2c2b30292144fa89006b1f9cb996876a38d35804b61f4834f89 Oct 30 03:43:14.723678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.723645 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:43:14.723781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.723712 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:43:14.723781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:14.723743 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:43:15.501089 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:15.501050 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" event={"ID":"7554b9fe-6f88-48a3-bc9d-4891493ad5a9","Type":"ContainerStarted","Data":"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5"} Oct 30 03:43:15.501637 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:15.501133 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" event={"ID":"7554b9fe-6f88-48a3-bc9d-4891493ad5a9","Type":"ContainerStarted","Data":"f45cf3e4a8b3f2c2b30292144fa89006b1f9cb996876a38d35804b61f4834f89"} Oct 30 03:43:15.606726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:15.606663 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podStartSLOduration=1.606648867 podStartE2EDuration="1.606648867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:15.605061481 +0000 UTC m=+2423.590181451" watchObservedRunningTime="2026-10-30 03:43:15.606648867 +0000 UTC m=+2423.591768824" Oct 30 03:43:16.398584 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:16.398551 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:16.400041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:16.400015 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 03:43:16.400152 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:16.400063 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 03:43:17.398953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:17.398906 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 03:43:17.399478 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:17.398976 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 03:43:18.398721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.398685 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 03:43:18.398903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.398744 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 03:43:18.775210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.775167 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:43:18.779658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.779633 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.794930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.794901 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:43:18.807550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.803343 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"agc-cred-70a168d6ae2f\"" Oct 30 03:43:18.807550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.803343 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"payload-processing-external-model-e2e--1a067c9a-dockercfg-rknrm\"" Oct 30 03:43:18.847888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.847852 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"payload-processing-external-model-plugins-e2e-extmod-b-855e14\"" Oct 30 03:43:18.848097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.847886 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"openshift-service-ca.crt\"" Oct 30 03:43:18.848097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.847893 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"routing-overlay\"" Oct 30 03:43:18.848097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.847916 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-855e14\"/\"kube-root-ca.crt\"" Oct 30 03:43:18.887668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.887637 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.887841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.887796 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.887932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.887869 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhh6k\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.888018 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.888003 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.989276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.989238 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.989435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.989346 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.989435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.989374 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhh6k\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.989557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.989441 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.990136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.990106 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.990236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.990120 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:18.991942 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:18.991920 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:19.008648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.008621 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhh6k\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\") pod \"payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:19.095233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.095139 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:19.398210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.398131 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 03:43:19.398210 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.398188 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 03:43:19.418632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.418604 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:43:19.419953 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:43:19.419924 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031ccd0a_4459_4d6a_ad3b_50940cbb0efb.slice/crio-aa93b8d8035917efca1c3ee13d577963f2a8f62933b81947fbdfa435c54149fd WatchSource:0}: Error finding container aa93b8d8035917efca1c3ee13d577963f2a8f62933b81947fbdfa435c54149fd: Status 404 returned error can't find the container with id aa93b8d8035917efca1c3ee13d577963f2a8f62933b81947fbdfa435c54149fd Oct 30 03:43:19.526554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:19.526511 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" event={"ID":"031ccd0a-4459-4d6a-ad3b-50940cbb0efb","Type":"ContainerStarted","Data":"aa93b8d8035917efca1c3ee13d577963f2a8f62933b81947fbdfa435c54149fd"} Oct 30 03:43:20.398723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:20.398692 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 03:43:20.399109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:20.398745 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 03:43:20.533636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:20.533597 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" event={"ID":"031ccd0a-4459-4d6a-ad3b-50940cbb0efb","Type":"ContainerStarted","Data":"82a54b61686250dc8900b7b36eab32d50689bb77e17a18c34d1696791001ffa7"} Oct 30 03:43:20.533823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:20.533791 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:20.579055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:20.578986 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" podStartSLOduration=2.578961691 podStartE2EDuration="2.578961691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:20.574876718 +0000 UTC m=+2428.559996627" watchObservedRunningTime="2026-10-30 03:43:20.578961691 +0000 UTC m=+2428.564081580" Oct 30 03:43:21.407957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:21.407914 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:21.408617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:21.408332 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:21.409457 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:21.409435 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 03:43:31.544076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:43:31.544035 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:43:41.748931 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:43:41.748886 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:44:09.084989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.084782 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:44:09.093001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.092967 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.105292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.105261 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:44:09.114614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.114587 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\"" Oct 30 03:44:09.114851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.114692 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\"" Oct 30 03:44:09.114851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.114674 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-0e64cda2603e\"" Oct 30 03:44:09.114851 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.114743 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\"" Oct 30 03:44:09.120403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.120382 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\"" Oct 30 03:44:09.120403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.120393 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-7e237115c4ea\"" Oct 30 03:44:09.120566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.120399 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-t5k2w\"" Oct 30 03:44:09.156598 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.156566 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.156771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.156613 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.156771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.156679 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxjq\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.156771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.156723 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.179959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.179927 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 03:44:09.184594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.184572 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.205919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.205884 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 03:44:09.208052 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.208028 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-441461-serving-cert\"" Oct 30 03:44:09.257323 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.257290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.257696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.257670 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.257812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.257785 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvlg\" (UniqueName: \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.257964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.257947 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.258070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258053 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.258135 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258121 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.258195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258168 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.258234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258216 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.258359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258331 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxjq\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.258903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.258879 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.260730 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.260709 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.277741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.277707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxjq\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\") pod \"payload-processing-external-model-e2e-extmod-b-372480-566fhztbm\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.359546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.359430 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvlg\" (UniqueName: \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.359700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.359545 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.359700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.359609 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.359812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.359696 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.360305 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.360280 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.362228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.362205 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.362307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.362213 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.378287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.378261 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvlg\" (UniqueName: \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\") pod \"maas-api-e2e-extmod-a-441461-788748b7-hmhdk\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.408273 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.408244 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.504865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.504833 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:09.713386 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.713350 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:44:09.713805 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:44:09.713781 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602654da_a6d1_40df_a05a_64f50a7879a0.slice/crio-0bccd7390080aac07865c9fc067fb126781786947b447155d3f91e836e0d4e2e WatchSource:0}: Error finding container 0bccd7390080aac07865c9fc067fb126781786947b447155d3f91e836e0d4e2e: Status 404 returned error can't find the container with id 0bccd7390080aac07865c9fc067fb126781786947b447155d3f91e836e0d4e2e Oct 30 03:44:09.809723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.809694 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 03:44:09.810387 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:44:09.810360 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bc333c_d0dd_4969_9d87_5836d80358ee.slice/crio-0a86d71e00a8c419602743f3bd332951260779dcad1386ceca14d72148e19503 WatchSource:0}: Error finding container 0a86d71e00a8c419602743f3bd332951260779dcad1386ceca14d72148e19503: Status 404 returned error can't find the container with id 0a86d71e00a8c419602743f3bd332951260779dcad1386ceca14d72148e19503 Oct 30 03:44:09.837784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.837751 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" event={"ID":"96bc333c-d0dd-4969-9d87-5836d80358ee","Type":"ContainerStarted","Data":"0a86d71e00a8c419602743f3bd332951260779dcad1386ceca14d72148e19503"} Oct 30 03:44:09.839374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.839344 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" event={"ID":"602654da-a6d1-40df-a05a-64f50a7879a0","Type":"ContainerStarted","Data":"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d"} Oct 30 03:44:09.839538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.839387 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" event={"ID":"602654da-a6d1-40df-a05a-64f50a7879a0","Type":"ContainerStarted","Data":"0bccd7390080aac07865c9fc067fb126781786947b447155d3f91e836e0d4e2e"} Oct 30 03:44:09.839538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.839498 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:09.881008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:09.880958 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" podStartSLOduration=0.880944006 podStartE2EDuration="880.944006ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:09.87957133 +0000 UTC m=+2477.864691220" watchObservedRunningTime="2026-10-30 03:44:09.880944006 +0000 UTC m=+2477.866063884" Oct 30 03:44:10.848053 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:10.848016 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" event={"ID":"96bc333c-d0dd-4969-9d87-5836d80358ee","Type":"ContainerStarted","Data":"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7"} Oct 30 03:44:10.848501 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:10.848159 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:10.894437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:10.894384 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" podStartSLOduration=1.517521116 podStartE2EDuration="1.894369642s" totalImagesPullingTime="376.848526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:10.891865725 +0000 UTC m=+2478.876985630" watchObservedRunningTime="2026-10-30 03:44:10.894369642 +0000 UTC m=+2478.879489529" Oct 30 03:44:11.749488 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:44:11.749453 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:44:16.860387 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:16.860356 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 03:44:20.851906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:20.851877 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:44:41.750582 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:44:41.750539 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:44:47.906452 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:47.906407 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:44:52.907456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:52.907402 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" podUID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" containerName="payload-processing" containerID="cri-o://82a54b61686250dc8900b7b36eab32d50689bb77e17a18c34d1696791001ffa7" gracePeriod=25 Oct 30 03:44:53.110683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.110645 4560 generic.go:356] "Generic (PLEG): container finished" podID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" containerID="82a54b61686250dc8900b7b36eab32d50689bb77e17a18c34d1696791001ffa7" exitCode=0 Oct 30 03:44:53.110897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.110849 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" event={"ID":"031ccd0a-4459-4d6a-ad3b-50940cbb0efb","Type":"ContainerDied","Data":"82a54b61686250dc8900b7b36eab32d50689bb77e17a18c34d1696791001ffa7"} Oct 30 03:44:53.170532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.170453 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:44:53.293628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.293587 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\") pod \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " Oct 30 03:44:53.293843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.293693 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\") pod \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " Oct 30 03:44:53.293843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.293760 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\") pod \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " Oct 30 03:44:53.293976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.293866 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\") pod \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\" (UID: \"031ccd0a-4459-4d6a-ad3b-50940cbb0efb\") " Oct 30 03:44:53.294091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.294065 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume" pod "031ccd0a-4459-4d6a-ad3b-50940cbb0efb" (UID: "031ccd0a-4459-4d6a-ad3b-50940cbb0efb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:53.294159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.294137 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume" pod "031ccd0a-4459-4d6a-ad3b-50940cbb0efb" (UID: "031ccd0a-4459-4d6a-ad3b-50940cbb0efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:53.294310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.294295 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:44:53.294364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.294312 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:44:53.296179 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.296151 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k" pod "031ccd0a-4459-4d6a-ad3b-50940cbb0efb" (UID: "031ccd0a-4459-4d6a-ad3b-50940cbb0efb"). InnerVolumeSpecName "kube-api-access-rhh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:53.296294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.296180 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials" pod "031ccd0a-4459-4d6a-ad3b-50940cbb0efb" (UID: "031ccd0a-4459-4d6a-ad3b-50940cbb0efb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:53.395972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.395939 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhh6k\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-kube-api-access-rhh6k\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:44:53.395972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:53.395966 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/031ccd0a-4459-4d6a-ad3b-50940cbb0efb-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.121913 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:54.121873 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" event={"ID":"031ccd0a-4459-4d6a-ad3b-50940cbb0efb","Type":"ContainerDied","Data":"aa93b8d8035917efca1c3ee13d577963f2a8f62933b81947fbdfa435c54149fd"} Oct 30 03:44:54.122411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:54.121948 4560 scope.go:122] "RemoveContainer" containerID="82a54b61686250dc8900b7b36eab32d50689bb77e17a18c34d1696791001ffa7" Oct 30 03:44:54.122411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:54.121898 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw" Oct 30 03:44:54.172096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:54.172064 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:44:54.186315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:54.186282 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-855e14/payload-processing-external-model-e2e-extmod-b-855e14-6fb84qdhw"] Oct 30 03:44:56.152736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:56.152663 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" path="/var/lib/kubelet/pods/031ccd0a-4459-4d6a-ad3b-50940cbb0efb/volumes" Oct 30 03:44:59.090213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.090176 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 03:44:59.090898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.090880 4560 cpu_manager.go:414] "containerMap: removing container" podUID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" containerName="payload-processing" Oct 30 03:44:59.090970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.090900 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" containerName="payload-processing" Oct 30 03:44:59.091077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.091063 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="031ccd0a-4459-4d6a-ad3b-50940cbb0efb" containerName="payload-processing" Oct 30 03:44:59.096067 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.096044 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.112647 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.112620 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 03:44:59.121966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.121945 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-81afe3\"" Oct 30 03:44:59.122190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.122175 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-81afe3-dockercfg-2dkhg\"" Oct 30 03:44:59.165036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.164999 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qsx\" (UniqueName: \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.165195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.165056 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.202564 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.202492 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 03:44:59.208164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.208135 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.224854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.224821 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 03:44:59.265859 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.265821 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qsx\" (UniqueName: \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.266036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.265899 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.266627 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.266608 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.285070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.285042 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qsx\" (UniqueName: \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\") pod \"payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.367634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.367545 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.367634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.367628 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn44x\" (UniqueName: \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.410541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.410484 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:44:59.469725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.469393 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.469725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.469444 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn44x\" (UniqueName: \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.470436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.470405 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.489303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.489252 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn44x\" (UniqueName: \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\") pod \"payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.525723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.525696 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:44:59.716630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.716596 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 03:44:59.720662 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:44:59.720633 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85978bcc_40fa_4f49_b594_e6c5061b4ae1.slice/crio-b88b87e00f68453df496aaed181a79078c55d64caca13c220c157bb84e485067 WatchSource:0}: Error finding container b88b87e00f68453df496aaed181a79078c55d64caca13c220c157bb84e485067: Status 404 returned error can't find the container with id b88b87e00f68453df496aaed181a79078c55d64caca13c220c157bb84e485067 Oct 30 03:44:59.828377 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:44:59.828339 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b0fe91_c609_4cbb_a99e_165aaf07a294.slice/crio-8ea09928c663e2ebebe9cd8ae9da9e393403480f0457f5691bea82c727c09339 WatchSource:0}: Error finding container 8ea09928c663e2ebebe9cd8ae9da9e393403480f0457f5691bea82c727c09339: Status 404 returned error can't find the container with id 8ea09928c663e2ebebe9cd8ae9da9e393403480f0457f5691bea82c727c09339 Oct 30 03:44:59.828531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:44:59.828490 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 03:45:00.165401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.165300 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" event={"ID":"c1b0fe91-c609-4cbb-a99e-165aaf07a294","Type":"ContainerStarted","Data":"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6"} Oct 30 03:45:00.165401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.165368 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" event={"ID":"c1b0fe91-c609-4cbb-a99e-165aaf07a294","Type":"ContainerStarted","Data":"8ea09928c663e2ebebe9cd8ae9da9e393403480f0457f5691bea82c727c09339"} Oct 30 03:45:00.165951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.165550 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:45:00.167568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.167507 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" event={"ID":"85978bcc-40fa-4f49-b594-e6c5061b4ae1","Type":"ContainerStarted","Data":"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86"} Oct 30 03:45:00.167721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.167589 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" event={"ID":"85978bcc-40fa-4f49-b594-e6c5061b4ae1","Type":"ContainerStarted","Data":"b88b87e00f68453df496aaed181a79078c55d64caca13c220c157bb84e485067"} Oct 30 03:45:00.167721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.167641 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:45:00.195554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.192904 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:00.199641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.199046 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.217896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.216265 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:00.242594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.224692 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" podStartSLOduration=1.224673141 podStartE2EDuration="1.224673141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:00.212114868 +0000 UTC m=+2528.197234757" watchObservedRunningTime="2026-10-30 03:45:00.224673141 +0000 UTC m=+2528.209793042" Oct 30 03:45:00.242594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.226081 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:00.242594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.228898 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.242594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.232872 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:00.242594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.232982 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.251543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.251382 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:00.256752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.256727 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:00.256916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.256857 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.270132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.269306 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:00.272088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.271916 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:45:00.287995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.286771 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:00.288963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.288931 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.289150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.289133 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ljm\" (UniqueName: \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.318293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.318236 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" podStartSLOduration=1.3182205200000001 podStartE2EDuration="1.31822052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:00.316063079 +0000 UTC m=+2528.301182969" watchObservedRunningTime="2026-10-30 03:45:00.31822052 +0000 UTC m=+2528.303340407" Oct 30 03:45:00.389910 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.389876 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.390088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.389921 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.390088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.389997 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw9d\" (UniqueName: \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.390172 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390096 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.390208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390169 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lq4\" (UniqueName: \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.390328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390304 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.390364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390344 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ljm\" (UniqueName: \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.390402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390373 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxqp\" (UniqueName: \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.391003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.390985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.409429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.409399 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ljm\" (UniqueName: \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.491685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491647 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxqp\" (UniqueName: \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.491882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491694 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.491882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491736 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.491882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491768 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw9d\" (UniqueName: \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.491882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491845 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.492065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.491928 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lq4\" (UniqueName: \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.492653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.492631 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.492758 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.492653 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.492808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.492791 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.511161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.511134 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw9d\" (UniqueName: \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.511267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.511134 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxqp\" (UniqueName: \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.511267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.511191 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lq4\" (UniqueName: \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.525970 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.525947 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:00.563399 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.563352 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:00.582068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.582034 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:00.599657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.595683 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:00.834825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.834796 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:00.835476 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:00.835443 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88539de_8465_48a9_8d8c_4840f361e623.slice/crio-704d3d98c2cb0ae98df7ea6b1c32ac5203cb8a39f8e9f41617d4cb767c378d5f WatchSource:0}: Error finding container 704d3d98c2cb0ae98df7ea6b1c32ac5203cb8a39f8e9f41617d4cb767c378d5f: Status 404 returned error can't find the container with id 704d3d98c2cb0ae98df7ea6b1c32ac5203cb8a39f8e9f41617d4cb767c378d5f Oct 30 03:45:00.889879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.889846 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:00.890440 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:00.890412 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2e4f94_9168_4b5a_ac63_9fee6d114352.slice/crio-a08484b1eb9290ba839a197119ed55f3c56f72e724baf623b92cc9d5d2b5fc05 WatchSource:0}: Error finding container a08484b1eb9290ba839a197119ed55f3c56f72e724baf623b92cc9d5d2b5fc05: Status 404 returned error can't find the container with id a08484b1eb9290ba839a197119ed55f3c56f72e724baf623b92cc9d5d2b5fc05 Oct 30 03:45:00.919105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.919076 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:00.922747 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:00.922718 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c94b0d9_1861_442c_bda4_b5ccfb35cc24.slice/crio-3802ab5ba5534d883433f4812bca0a88f183b01dd00e5d68646cd651f81623ee WatchSource:0}: Error finding container 3802ab5ba5534d883433f4812bca0a88f183b01dd00e5d68646cd651f81623ee: Status 404 returned error can't find the container with id 3802ab5ba5534d883433f4812bca0a88f183b01dd00e5d68646cd651f81623ee Oct 30 03:45:00.945552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:00.945489 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:00.945664 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:00.945555 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c08a1c_aa8b_4d6e_98cd_21fd050a50d1.slice/crio-4cf7ac9795ccf50ce41614ea15cbaf776d574d4b8255198a7b638e3047b0ab3b WatchSource:0}: Error finding container 4cf7ac9795ccf50ce41614ea15cbaf776d574d4b8255198a7b638e3047b0ab3b: Status 404 returned error can't find the container with id 4cf7ac9795ccf50ce41614ea15cbaf776d574d4b8255198a7b638e3047b0ab3b Oct 30 03:45:01.174578 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.174542 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" event={"ID":"b88539de-8465-48a9-8d8c-4840f361e623","Type":"ContainerStarted","Data":"704d3d98c2cb0ae98df7ea6b1c32ac5203cb8a39f8e9f41617d4cb767c378d5f"} Oct 30 03:45:01.176426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.176395 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerStarted","Data":"3802ab5ba5534d883433f4812bca0a88f183b01dd00e5d68646cd651f81623ee"} Oct 30 03:45:01.178055 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.178027 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerStarted","Data":"d64af13071ce94b01cd9ecc8a8b7e5c2c0ff43937f6628d8cedaf91295b2518f"} Oct 30 03:45:01.178180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.178074 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerStarted","Data":"a08484b1eb9290ba839a197119ed55f3c56f72e724baf623b92cc9d5d2b5fc05"} Oct 30 03:45:01.179328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.179301 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerStarted","Data":"4cf7ac9795ccf50ce41614ea15cbaf776d574d4b8255198a7b638e3047b0ab3b"} Oct 30 03:45:01.219714 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:01.219660 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" podStartSLOduration=1.219645848 podStartE2EDuration="1.219645848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:01.217121121 +0000 UTC m=+2529.202241027" watchObservedRunningTime="2026-10-30 03:45:01.219645848 +0000 UTC m=+2529.204765735" Oct 30 03:45:02.187117 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.187025 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerID="d64af13071ce94b01cd9ecc8a8b7e5c2c0ff43937f6628d8cedaf91295b2518f" exitCode=60 Oct 30 03:45:02.187592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.187108 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerDied","Data":"d64af13071ce94b01cd9ecc8a8b7e5c2c0ff43937f6628d8cedaf91295b2518f"} Oct 30 03:45:02.187592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.187385 4560 scope.go:122] "RemoveContainer" containerID="d64af13071ce94b01cd9ecc8a8b7e5c2c0ff43937f6628d8cedaf91295b2518f" Oct 30 03:45:02.189044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.189021 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerStarted","Data":"672cb1c1cbe922928d6b63a047f79b68c0a080b53137799d37e013d83fe80f90"} Oct 30 03:45:02.190926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.190906 4560 generic.go:356] "Generic (PLEG): container finished" podID="b88539de-8465-48a9-8d8c-4840f361e623" containerID="865f9d23643ed305558798082c371804525fbea2b5d9861c6f992d11c7862c39" exitCode=60 Oct 30 03:45:02.191027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.191008 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" event={"ID":"b88539de-8465-48a9-8d8c-4840f361e623","Type":"ContainerDied","Data":"865f9d23643ed305558798082c371804525fbea2b5d9861c6f992d11c7862c39"} Oct 30 03:45:02.191250 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.191229 4560 scope.go:122] "RemoveContainer" containerID="865f9d23643ed305558798082c371804525fbea2b5d9861c6f992d11c7862c39" Oct 30 03:45:02.192812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.192791 4560 generic.go:356] "Generic (PLEG): container finished" podID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerID="7235cebe0b1999d2494cc4b78baf889d1a720287e1d7d4f781fff354c0059dde" exitCode=60 Oct 30 03:45:02.192897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.192868 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerDied","Data":"7235cebe0b1999d2494cc4b78baf889d1a720287e1d7d4f781fff354c0059dde"} Oct 30 03:45:02.193095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.193078 4560 scope.go:122] "RemoveContainer" containerID="7235cebe0b1999d2494cc4b78baf889d1a720287e1d7d4f781fff354c0059dde" Oct 30 03:45:02.357072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:02.357006 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" podStartSLOduration=2.35698698 podStartE2EDuration="2.35698698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:02.353073379 +0000 UTC m=+2530.338193267" watchObservedRunningTime="2026-10-30 03:45:02.35698698 +0000 UTC m=+2530.342106869" Oct 30 03:45:03.201947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.201909 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerStarted","Data":"14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb"} Oct 30 03:45:03.203875 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.203851 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerID="579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35" exitCode=60 Oct 30 03:45:03.204027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.203936 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerDied","Data":"579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35"} Oct 30 03:45:03.204027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.204016 4560 scope.go:122] "RemoveContainer" containerID="d64af13071ce94b01cd9ecc8a8b7e5c2c0ff43937f6628d8cedaf91295b2518f" Oct 30 03:45:03.204206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.204190 4560 scope.go:122] "RemoveContainer" containerID="579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35" Oct 30 03:45:03.204470 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:03.204449 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw_odh-ai-gateway-infra(bb2e4f94-9168-4b5a-ac63-9fee6d114352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" Oct 30 03:45:03.206048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.206028 4560 generic.go:356] "Generic (PLEG): container finished" podID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerID="672cb1c1cbe922928d6b63a047f79b68c0a080b53137799d37e013d83fe80f90" exitCode=60 Oct 30 03:45:03.206151 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.206111 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerDied","Data":"672cb1c1cbe922928d6b63a047f79b68c0a080b53137799d37e013d83fe80f90"} Oct 30 03:45:03.206309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.206296 4560 scope.go:122] "RemoveContainer" containerID="672cb1c1cbe922928d6b63a047f79b68c0a080b53137799d37e013d83fe80f90" Oct 30 03:45:03.208069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.208052 4560 generic.go:356] "Generic (PLEG): container finished" podID="b88539de-8465-48a9-8d8c-4840f361e623" containerID="dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c" exitCode=60 Oct 30 03:45:03.208130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.208079 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" event={"ID":"b88539de-8465-48a9-8d8c-4840f361e623","Type":"ContainerDied","Data":"dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c"} Oct 30 03:45:03.208311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.208297 4560 scope.go:122] "RemoveContainer" containerID="dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c" Oct 30 03:45:03.209004 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:03.208977 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7_odh-ai-gateway-infra(b88539de-8465-48a9-8d8c-4840f361e623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" podUID="b88539de-8465-48a9-8d8c-4840f361e623" Oct 30 03:45:03.219742 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.219713 4560 scope.go:122] "RemoveContainer" containerID="865f9d23643ed305558798082c371804525fbea2b5d9861c6f992d11c7862c39" Oct 30 03:45:03.244495 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:03.244444 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" podStartSLOduration=3.24442597 podStartE2EDuration="3.24442597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.242511262 +0000 UTC m=+2531.227631150" watchObservedRunningTime="2026-10-30 03:45:03.24442597 +0000 UTC m=+2531.229545884" Oct 30 03:45:04.218982 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.218947 4560 generic.go:356] "Generic (PLEG): container finished" podID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerID="1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed" exitCode=60 Oct 30 03:45:04.219438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.219027 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerDied","Data":"1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed"} Oct 30 03:45:04.219438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.219084 4560 scope.go:122] "RemoveContainer" containerID="672cb1c1cbe922928d6b63a047f79b68c0a080b53137799d37e013d83fe80f90" Oct 30 03:45:04.219438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.219309 4560 scope.go:122] "RemoveContainer" containerID="1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed" Oct 30 03:45:04.219734 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:04.219703 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv_odh-ai-gateway-infra(00c08a1c-aa8b-4d6e-98cd-21fd050a50d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" Oct 30 03:45:04.221281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.221265 4560 scope.go:122] "RemoveContainer" containerID="dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c" Oct 30 03:45:04.221570 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:04.221548 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7_odh-ai-gateway-infra(b88539de-8465-48a9-8d8c-4840f361e623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" podUID="b88539de-8465-48a9-8d8c-4840f361e623" Oct 30 03:45:04.222880 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.222860 4560 generic.go:356] "Generic (PLEG): container finished" podID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerID="14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb" exitCode=60 Oct 30 03:45:04.222976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.222944 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerDied","Data":"14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb"} Oct 30 03:45:04.223115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.223098 4560 scope.go:122] "RemoveContainer" containerID="14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb" Oct 30 03:45:04.223346 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:04.223328 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr_odh-ai-gateway-infra(3c94b0d9-1861-442c-bda4-b5ccfb35cc24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" Oct 30 03:45:04.224769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.224751 4560 scope.go:122] "RemoveContainer" containerID="579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35" Oct 30 03:45:04.224996 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:04.224978 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw_odh-ai-gateway-infra(bb2e4f94-9168-4b5a-ac63-9fee6d114352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" Oct 30 03:45:04.232997 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.232977 4560 scope.go:122] "RemoveContainer" containerID="7235cebe0b1999d2494cc4b78baf889d1a720287e1d7d4f781fff354c0059dde" Oct 30 03:45:04.373249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.373163 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:04.380198 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.380173 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.395276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.395244 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:04.399020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.398986 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"kube-root-ca.crt\"" Oct 30 03:45:04.399020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.398998 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"routing-overlay\"" Oct 30 03:45:04.399020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.399011 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"payload-processing-external-model-plugins-e2e-extmod-a-398c25\"" Oct 30 03:45:04.399215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.399015 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"payload-processing-external-model-e2e--dda521a1-dockercfg-f6fwp\"" Oct 30 03:45:04.399215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.398998 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"agc-cred-5f185e7f1678\"" Oct 30 03:45:04.399215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.399154 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"openshift-service-ca.crt\"" Oct 30 03:45:04.545510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.545462 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.545722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.545619 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.545722 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.545702 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.545896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.545867 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6qpt\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.647737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.647655 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.647737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.647712 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.647938 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.647761 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6qpt\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.647979 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.647942 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.648464 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.648441 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.648531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.648499 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.650219 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.650195 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.667910 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.667880 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6qpt\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.695267 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.695241 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:04.955201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.955166 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:45:04.961204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.961181 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:04.977770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.977747 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-398c25\"/\"agc-cred-d1428e5f4fde\"" Oct 30 03:45:04.980216 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:04.980191 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:45:05.010650 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.010610 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:05.015991 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:05.015963 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24585f36_2893_48fe_b917_487dafd6fe88.slice/crio-5154f03d620d3774157f1ac5ee7b1e84da7b07048e2d60bf3cd7254b7ec58369 WatchSource:0}: Error finding container 5154f03d620d3774157f1ac5ee7b1e84da7b07048e2d60bf3cd7254b7ec58369: Status 404 returned error can't find the container with id 5154f03d620d3774157f1ac5ee7b1e84da7b07048e2d60bf3cd7254b7ec58369 Oct 30 03:45:05.156483 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.156447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.156679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.156628 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.156679 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.156657 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrcp\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.156840 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.156818 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.232353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.232258 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" event={"ID":"24585f36-2893-48fe-b917-487dafd6fe88","Type":"ContainerStarted","Data":"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb"} Oct 30 03:45:05.232353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.232312 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" event={"ID":"24585f36-2893-48fe-b917-487dafd6fe88","Type":"ContainerStarted","Data":"5154f03d620d3774157f1ac5ee7b1e84da7b07048e2d60bf3cd7254b7ec58369"} Oct 30 03:45:05.232353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.232340 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:05.234216 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.234197 4560 scope.go:122] "RemoveContainer" containerID="1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed" Oct 30 03:45:05.234569 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:05.234506 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv_odh-ai-gateway-infra(00c08a1c-aa8b-4d6e-98cd-21fd050a50d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" Oct 30 03:45:05.235887 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.235868 4560 scope.go:122] "RemoveContainer" containerID="14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb" Oct 30 03:45:05.236105 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:05.236087 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr_odh-ai-gateway-infra(3c94b0d9-1861-442c-bda4-b5ccfb35cc24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" Oct 30 03:45:05.258406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.258381 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.258406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.258410 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrcp\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.258596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.258474 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.258596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.258557 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.259205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.259179 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.259283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.259218 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.261270 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.261250 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.271721 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.271667 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" podStartSLOduration=1.271651943 podStartE2EDuration="1.271651943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:05.27023385 +0000 UTC m=+2533.255353796" watchObservedRunningTime="2026-10-30 03:45:05.271651943 +0000 UTC m=+2533.256771833" Oct 30 03:45:05.280837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.280813 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrcp\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-8565697sc\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.576592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.576477 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:05.882153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:05.882122 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:45:05.882562 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:05.882509 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603eb620_b75e_4c84_b172_96d94900f090.slice/crio-ea7a3245162b27287f3833c89f4d06e5153437c8c8e351abd3c39ac36745e55d WatchSource:0}: Error finding container ea7a3245162b27287f3833c89f4d06e5153437c8c8e351abd3c39ac36745e55d: Status 404 returned error can't find the container with id ea7a3245162b27287f3833c89f4d06e5153437c8c8e351abd3c39ac36745e55d Oct 30 03:45:06.243903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:06.243869 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" event={"ID":"603eb620-b75e-4c84-b172-96d94900f090","Type":"ContainerStarted","Data":"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7"} Oct 30 03:45:06.244337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:06.243931 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" event={"ID":"603eb620-b75e-4c84-b172-96d94900f090","Type":"ContainerStarted","Data":"ea7a3245162b27287f3833c89f4d06e5153437c8c8e351abd3c39ac36745e55d"} Oct 30 03:45:06.244337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:06.244191 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:06.286150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:06.286097 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" podStartSLOduration=2.286083375 podStartE2EDuration="2.286083375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:06.28276781 +0000 UTC m=+2534.267887708" watchObservedRunningTime="2026-10-30 03:45:06.286083375 +0000 UTC m=+2534.271203263" Oct 30 03:45:11.182534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:11.182494 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 03:45:11.182922 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:11.182681 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 03:45:11.749613 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:11.749569 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:45:14.145213 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:14.145175 4560 scope.go:122] "RemoveContainer" containerID="dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c" Oct 30 03:45:14.145708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:14.145265 4560 scope.go:122] "RemoveContainer" containerID="579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35" Oct 30 03:45:15.145236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.145203 4560 scope.go:122] "RemoveContainer" containerID="14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb" Oct 30 03:45:15.145690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.145318 4560 scope.go:122] "RemoveContainer" containerID="1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed" Oct 30 03:45:15.307272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.307198 4560 generic.go:356] "Generic (PLEG): container finished" podID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerID="535d3b26b7562a4a5baf1a3576c50ae9da4fe0b1b98a2ccefc0f6fb67a41d33c" exitCode=60 Oct 30 03:45:15.307411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.307280 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerDied","Data":"535d3b26b7562a4a5baf1a3576c50ae9da4fe0b1b98a2ccefc0f6fb67a41d33c"} Oct 30 03:45:15.307411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.307347 4560 scope.go:122] "RemoveContainer" containerID="579eb26358b9733b921a0f17071a4a189a2a0477bdc8779e500c70b784998f35" Oct 30 03:45:15.307755 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.307730 4560 scope.go:122] "RemoveContainer" containerID="535d3b26b7562a4a5baf1a3576c50ae9da4fe0b1b98a2ccefc0f6fb67a41d33c" Oct 30 03:45:15.308053 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:15.308021 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw_odh-ai-gateway-infra(bb2e4f94-9168-4b5a-ac63-9fee6d114352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" Oct 30 03:45:15.309413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.309394 4560 generic.go:356] "Generic (PLEG): container finished" podID="b88539de-8465-48a9-8d8c-4840f361e623" containerID="ae089b8705587809b3681cfda72c59da3ca0c8a6251e46e00a4477abeab7925e" exitCode=60 Oct 30 03:45:15.309537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.309473 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" event={"ID":"b88539de-8465-48a9-8d8c-4840f361e623","Type":"ContainerDied","Data":"ae089b8705587809b3681cfda72c59da3ca0c8a6251e46e00a4477abeab7925e"} Oct 30 03:45:15.309854 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.309837 4560 scope.go:122] "RemoveContainer" containerID="ae089b8705587809b3681cfda72c59da3ca0c8a6251e46e00a4477abeab7925e" Oct 30 03:45:15.310098 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:15.310078 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7_odh-ai-gateway-infra(b88539de-8465-48a9-8d8c-4840f361e623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" podUID="b88539de-8465-48a9-8d8c-4840f361e623" Oct 30 03:45:15.321008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:15.320985 4560 scope.go:122] "RemoveContainer" containerID="dbadf98eec8eb60db8ca06fece7ed2c268b275de6f42e9fc98f403453a456a8c" Oct 30 03:45:16.248083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.248051 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:16.319656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.319572 4560 generic.go:356] "Generic (PLEG): container finished" podID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerID="0e43970df9555ca218a5fb120a6842e3cf4f8d248d13340d365cc51f472ef00b" exitCode=60 Oct 30 03:45:16.319825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.319648 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerDied","Data":"0e43970df9555ca218a5fb120a6842e3cf4f8d248d13340d365cc51f472ef00b"} Oct 30 03:45:16.319825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.319718 4560 scope.go:122] "RemoveContainer" containerID="14d8f488a478e3f23aeb62a1cb89514daae0877745a86a84b711d040ace504fb" Oct 30 03:45:16.320182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.320159 4560 scope.go:122] "RemoveContainer" containerID="0e43970df9555ca218a5fb120a6842e3cf4f8d248d13340d365cc51f472ef00b" Oct 30 03:45:16.320505 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:16.320478 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr_odh-ai-gateway-infra(3c94b0d9-1861-442c-bda4-b5ccfb35cc24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" Oct 30 03:45:16.324217 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.324196 4560 generic.go:356] "Generic (PLEG): container finished" podID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerID="3f5f41ee8285332ee0ca7a7a7b8a30070dab3a4bee9e52b7812d75c9dcf2aa56" exitCode=60 Oct 30 03:45:16.324309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.324240 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerDied","Data":"3f5f41ee8285332ee0ca7a7a7b8a30070dab3a4bee9e52b7812d75c9dcf2aa56"} Oct 30 03:45:16.324643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.324624 4560 scope.go:122] "RemoveContainer" containerID="3f5f41ee8285332ee0ca7a7a7b8a30070dab3a4bee9e52b7812d75c9dcf2aa56" Oct 30 03:45:16.324966 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:16.324942 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv_odh-ai-gateway-infra(00c08a1c-aa8b-4d6e-98cd-21fd050a50d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" Oct 30 03:45:16.333176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.333158 4560 scope.go:122] "RemoveContainer" containerID="1c615d4ce0f4921eb576d83ecc2c58c48bce0279e9bdb0725614b8491ed4efed" Oct 30 03:45:16.364983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.364955 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:16.403625 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.403590 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:16.538480 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.538452 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:16.600816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.600784 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:16.603004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.602973 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\") pod \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " Oct 30 03:45:16.603201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.603182 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\" (UniqueName: \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\") pod \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\" (UID: \"bb2e4f94-9168-4b5a-ac63-9fee6d114352\") " Oct 30 03:45:16.603307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.603281 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca" pod "bb2e4f94-9168-4b5a-ac63-9fee6d114352" (UID: "bb2e4f94-9168-4b5a-ac63-9fee6d114352"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:16.603728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.603705 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2e4f94-9168-4b5a-ac63-9fee6d114352-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:16.605201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.605178 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp" pod "bb2e4f94-9168-4b5a-ac63-9fee6d114352" (UID: "bb2e4f94-9168-4b5a-ac63-9fee6d114352"). InnerVolumeSpecName "kube-api-access-hwxqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:16.704355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.704320 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\") pod \"b88539de-8465-48a9-8d8c-4840f361e623\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " Oct 30 03:45:16.704355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.704359 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\" (UniqueName: \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\") pod \"b88539de-8465-48a9-8d8c-4840f361e623\" (UID: \"b88539de-8465-48a9-8d8c-4840f361e623\") " Oct 30 03:45:16.704729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.704707 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca" pod "b88539de-8465-48a9-8d8c-4840f361e623" (UID: "b88539de-8465-48a9-8d8c-4840f361e623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:16.704885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.704873 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88539de-8465-48a9-8d8c-4840f361e623-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:16.704919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.704888 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxqp\" (UniqueName: \"kubernetes.io/projected/bb2e4f94-9168-4b5a-ac63-9fee6d114352-kube-api-access-hwxqp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:16.706433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.706413 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm" pod "b88539de-8465-48a9-8d8c-4840f361e623" (UID: "b88539de-8465-48a9-8d8c-4840f361e623"). InnerVolumeSpecName "kube-api-access-t4ljm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:16.805796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:16.805763 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ljm\" (UniqueName: \"kubernetes.io/projected/b88539de-8465-48a9-8d8c-4840f361e623-kube-api-access-t4ljm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:17.255269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.255242 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:45:17.321693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.321661 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:17.340334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.340307 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" Oct 30 03:45:17.340777 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.340310 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw" event={"ID":"bb2e4f94-9168-4b5a-ac63-9fee6d114352","Type":"ContainerDied","Data":"a08484b1eb9290ba839a197119ed55f3c56f72e724baf623b92cc9d5d2b5fc05"} Oct 30 03:45:17.340923 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.340843 4560 scope.go:122] "RemoveContainer" containerID="535d3b26b7562a4a5baf1a3576c50ae9da4fe0b1b98a2ccefc0f6fb67a41d33c" Oct 30 03:45:17.345242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.345167 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" event={"ID":"b88539de-8465-48a9-8d8c-4840f361e623","Type":"ContainerDied","Data":"704d3d98c2cb0ae98df7ea6b1c32ac5203cb8a39f8e9f41617d4cb767c378d5f"} Oct 30 03:45:17.348543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.345471 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7" Oct 30 03:45:17.356506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.356485 4560 scope.go:122] "RemoveContainer" containerID="ae089b8705587809b3681cfda72c59da3ca0c8a6251e46e00a4477abeab7925e" Oct 30 03:45:17.379823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.379795 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:17.409785 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.409751 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:17.415282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.415242 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888865-jm9mw"] Oct 30 03:45:17.422950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.422912 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:17.468575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.465624 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:17.477225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.477188 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888865-pgdq7"] Oct 30 03:45:17.568241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.568209 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:17.643540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.643498 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:17.717748 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.717714 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\") pod \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " Oct 30 03:45:17.717928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.717855 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\" (UniqueName: \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\") pod \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\" (UID: \"3c94b0d9-1861-442c-bda4-b5ccfb35cc24\") " Oct 30 03:45:17.718022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.718007 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\" (UniqueName: \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\") pod \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " Oct 30 03:45:17.718109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.718074 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca" pod "3c94b0d9-1861-442c-bda4-b5ccfb35cc24" (UID: "3c94b0d9-1861-442c-bda4-b5ccfb35cc24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:17.718162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.718107 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\") pod \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\" (UID: \"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1\") " Oct 30 03:45:17.718542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.718470 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca" pod "00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" (UID: "00c08a1c-aa8b-4d6e-98cd-21fd050a50d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:17.718696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.718555 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:17.720161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.720137 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4" pod "00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" (UID: "00c08a1c-aa8b-4d6e-98cd-21fd050a50d1"). InnerVolumeSpecName "kube-api-access-47lq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:17.720259 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.720184 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d" pod "3c94b0d9-1861-442c-bda4-b5ccfb35cc24" (UID: "3c94b0d9-1861-442c-bda4-b5ccfb35cc24"). InnerVolumeSpecName "kube-api-access-xpw9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:17.820188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.820097 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpw9d\" (UniqueName: \"kubernetes.io/projected/3c94b0d9-1861-442c-bda4-b5ccfb35cc24-kube-api-access-xpw9d\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:17.820188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.820125 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lq4\" (UniqueName: \"kubernetes.io/projected/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-kube-api-access-47lq4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:17.820188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:17.820139 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:18.153431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.153350 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88539de-8465-48a9-8d8c-4840f361e623" path="/var/lib/kubelet/pods/b88539de-8465-48a9-8d8c-4840f361e623/volumes" Oct 30 03:45:18.153860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.153841 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" path="/var/lib/kubelet/pods/bb2e4f94-9168-4b5a-ac63-9fee6d114352/volumes" Oct 30 03:45:18.354103 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.354073 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" Oct 30 03:45:18.354603 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.354118 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv" event={"ID":"00c08a1c-aa8b-4d6e-98cd-21fd050a50d1","Type":"ContainerDied","Data":"4cf7ac9795ccf50ce41614ea15cbaf776d574d4b8255198a7b638e3047b0ab3b"} Oct 30 03:45:18.354603 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.354184 4560 scope.go:122] "RemoveContainer" containerID="3f5f41ee8285332ee0ca7a7a7b8a30070dab3a4bee9e52b7812d75c9dcf2aa56" Oct 30 03:45:18.356759 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.356741 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" Oct 30 03:45:18.356865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.356738 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr" event={"ID":"3c94b0d9-1861-442c-bda4-b5ccfb35cc24","Type":"ContainerDied","Data":"3802ab5ba5534d883433f4812bca0a88f183b01dd00e5d68646cd651f81623ee"} Oct 30 03:45:18.364358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.364339 4560 scope.go:122] "RemoveContainer" containerID="0e43970df9555ca218a5fb120a6842e3cf4f8d248d13340d365cc51f472ef00b" Oct 30 03:45:18.416983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.416898 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:18.424337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.424297 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888865-9v6tr"] Oct 30 03:45:18.455725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.455087 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:18.459165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:18.459136 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888865-jl6wv"] Oct 30 03:45:20.156355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:20.156323 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" path="/var/lib/kubelet/pods/00c08a1c-aa8b-4d6e-98cd-21fd050a50d1/volumes" Oct 30 03:45:20.156882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:20.156838 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" path="/var/lib/kubelet/pods/3c94b0d9-1861-442c-bda4-b5ccfb35cc24/volumes" Oct 30 03:45:22.322275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.322233 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" podUID="24585f36-2893-48fe-b917-487dafd6fe88" containerName="payload-processing" containerID="cri-o://b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb" gracePeriod=25 Oct 30 03:45:22.596963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.596936 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:22.680146 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680109 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\") pod \"24585f36-2893-48fe-b917-487dafd6fe88\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " Oct 30 03:45:22.680350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680194 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\") pod \"24585f36-2893-48fe-b917-487dafd6fe88\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " Oct 30 03:45:22.680350 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680258 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\") pod \"24585f36-2893-48fe-b917-487dafd6fe88\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " Oct 30 03:45:22.680500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680476 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\") pod \"24585f36-2893-48fe-b917-487dafd6fe88\" (UID: \"24585f36-2893-48fe-b917-487dafd6fe88\") " Oct 30 03:45:22.680660 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680572 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume" pod "24585f36-2893-48fe-b917-487dafd6fe88" (UID: "24585f36-2893-48fe-b917-487dafd6fe88"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:22.680916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680896 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:22.680989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.680908 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume" pod "24585f36-2893-48fe-b917-487dafd6fe88" (UID: "24585f36-2893-48fe-b917-487dafd6fe88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:22.683252 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.683218 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt" pod "24585f36-2893-48fe-b917-487dafd6fe88" (UID: "24585f36-2893-48fe-b917-487dafd6fe88"). InnerVolumeSpecName "kube-api-access-v6qpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:22.683711 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.683410 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials" pod "24585f36-2893-48fe-b917-487dafd6fe88" (UID: "24585f36-2893-48fe-b917-487dafd6fe88"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:22.782138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.782089 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24585f36-2893-48fe-b917-487dafd6fe88-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:22.782138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.782130 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:22.782375 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:22.782148 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6qpt\" (UniqueName: \"kubernetes.io/projected/24585f36-2893-48fe-b917-487dafd6fe88-kube-api-access-v6qpt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:23.399370 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.399340 4560 generic.go:356] "Generic (PLEG): container finished" podID="24585f36-2893-48fe-b917-487dafd6fe88" containerID="b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb" exitCode=0 Oct 30 03:45:23.399370 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.399377 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" event={"ID":"24585f36-2893-48fe-b917-487dafd6fe88","Type":"ContainerDied","Data":"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb"} Oct 30 03:45:23.401752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.399382 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" Oct 30 03:45:23.401752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.399416 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4" event={"ID":"24585f36-2893-48fe-b917-487dafd6fe88","Type":"ContainerDied","Data":"5154f03d620d3774157f1ac5ee7b1e84da7b07048e2d60bf3cd7254b7ec58369"} Oct 30 03:45:23.401752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.399442 4560 scope.go:122] "RemoveContainer" containerID="b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb" Oct 30 03:45:23.412425 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.412383 4560 scope.go:122] "RemoveContainer" containerID="b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb" Oct 30 03:45:23.413076 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:23.413040 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb\": container with ID starting with b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb not found: ID does not exist" containerID="b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb" Oct 30 03:45:23.413190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.413086 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb"} err="failed to get container status \"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb\": rpc error: code = NotFound desc = could not find container \"b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb\": container with ID starting with b1d41e18d7b2efcce1bc0136096d8a35b67965e466d7b6c7a10e5a3bdd5ca1fb not found: ID does not exist" Oct 30 03:45:23.459630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.459596 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:23.471162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:23.471130 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468g5nr4"] Oct 30 03:45:24.154023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:24.153988 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24585f36-2893-48fe-b917-487dafd6fe88" path="/var/lib/kubelet/pods/24585f36-2893-48fe-b917-487dafd6fe88/volumes" Oct 30 03:45:33.802320 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.802280 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 03:45:33.803111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803087 4560 cpu_manager.go:414] "containerMap: removing container" podUID="24585f36-2893-48fe-b917-487dafd6fe88" containerName="payload-processing" Oct 30 03:45:33.803111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803112 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24585f36-2893-48fe-b917-487dafd6fe88" containerName="payload-processing" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803133 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803141 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803153 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803161 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803182 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803189 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803199 4560 cpu_manager.go:414] "containerMap: removing container" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803206 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803220 4560 cpu_manager.go:414] "containerMap: removing container" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803227 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803242 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803249 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803260 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803267 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803278 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.803283 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803284 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803298 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803305 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803339 4560 cpu_manager.go:414] "containerMap: removing container" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803346 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803371 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803378 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803562 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803577 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803589 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803600 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803608 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803617 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803625 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803638 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803653 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="24585f36-2893-48fe-b917-487dafd6fe88" containerName="payload-processing" Oct 30 03:45:33.804600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.803662 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:45:33.808924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.808901 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:33.841543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.840225 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-81afe3-serving-cert\"" Oct 30 03:45:33.845062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.845023 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 03:45:33.912743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.912704 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:33.912946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.912757 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkcv\" (UniqueName: \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:33.912946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.912880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:33.913091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:33.913068 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.016372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.015225 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.016372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.015339 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.016372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.015383 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkcv\" (UniqueName: \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.016372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.015473 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.021990 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.021958 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.024250 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.024205 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.031195 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.031167 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.041718 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.041692 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkcv\" (UniqueName: \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\") pod \"maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.123315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.123239 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:34.428734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.428701 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 03:45:34.429906 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:45:34.429868 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb39b4ec_77a8_4d8b_b7b8_606c8ffb6a21.slice/crio-3d8a159e1b545e5d8963271fd0c6831557e5dfbe48e370ee00c90cd8085ffa62 WatchSource:0}: Error finding container 3d8a159e1b545e5d8963271fd0c6831557e5dfbe48e370ee00c90cd8085ffa62: Status 404 returned error can't find the container with id 3d8a159e1b545e5d8963271fd0c6831557e5dfbe48e370ee00c90cd8085ffa62 Oct 30 03:45:34.470242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:34.470202 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" event={"ID":"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21","Type":"ContainerStarted","Data":"3d8a159e1b545e5d8963271fd0c6831557e5dfbe48e370ee00c90cd8085ffa62"} Oct 30 03:45:35.477882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:35.477844 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" event={"ID":"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21","Type":"ContainerStarted","Data":"c7bd0b6ab79327704aa52103bf4ec05c344b9ba5e4251e1f9e6e3dc42ad04776"} Oct 30 03:45:35.478298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:35.477995 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:35.518185 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:35.518130 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" podStartSLOduration=2.134182632 podStartE2EDuration="2.51811414s" totalImagesPullingTime="383.931508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:35.517167578 +0000 UTC m=+2563.502287468" watchObservedRunningTime="2026-10-30 03:45:35.51811414 +0000 UTC m=+2563.503234028" Oct 30 03:45:41.491170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:41.491142 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 03:45:41.749809 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:41.749731 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:45:45.256793 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.256753 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-0e64cda2603e: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:45.257388 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.257366 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:45.257467 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.257457 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials podName:602654da-a6d1-40df-a05a-64f50a7879a0 nodeName:}" failed. No retries permitted until 2026-10-30 03:45:45.757437476 +0000 UTC m=+2573.742557364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0") : secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:45.764327 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.764293 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-0e64cda2603e: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:45.764327 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.764326 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:45.764553 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:45.764413 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials podName:602654da-a6d1-40df-a05a-64f50a7879a0 nodeName:}" failed. No retries permitted until 2026-10-30 03:45:46.764396765 +0000 UTC m=+2574.749516630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0") : secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:46.774635 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:46.774599 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-0e64cda2603e: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:46.774635 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:46.774634 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:46.775083 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:46.774691 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials podName:602654da-a6d1-40df-a05a-64f50a7879a0 nodeName:}" failed. No retries permitted until 2026-10-30 03:45:48.774676793 +0000 UTC m=+2576.759796674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0") : secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:48.800097 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:48.800062 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-0e64cda2603e: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:48.800097 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:48.800089 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:48.800541 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:48.800141 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials podName:602654da-a6d1-40df-a05a-64f50a7879a0 nodeName:}" failed. No retries permitted until 2026-10-30 03:45:52.800127333 +0000 UTC m=+2580.785247223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0") : secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:52.850356 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:52.850319 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-0e64cda2603e: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:52.850356 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:52.850358 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm: secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:52.850893 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:52.850415 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials podName:602654da-a6d1-40df-a05a-64f50a7879a0 nodeName:}" failed. No retries permitted until 2026-10-30 03:46:00.85039955 +0000 UTC m=+2588.835519428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0") : secret "agc-cred-0e64cda2603e" not found Oct 30 03:45:53.107533 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:53.107416 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:45:58.108444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.108395 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" podUID="602654da-a6d1-40df-a05a-64f50a7879a0" containerName="payload-processing" containerID="cri-o://a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d" gracePeriod=25 Oct 30 03:45:58.374266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.374240 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:45:58.416935 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.416894 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\") pod \"602654da-a6d1-40df-a05a-64f50a7879a0\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " Oct 30 03:45:58.417132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.416953 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\") pod \"602654da-a6d1-40df-a05a-64f50a7879a0\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " Oct 30 03:45:58.417132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.417014 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\") pod \"602654da-a6d1-40df-a05a-64f50a7879a0\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " Oct 30 03:45:58.417132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.417048 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\") pod \"602654da-a6d1-40df-a05a-64f50a7879a0\" (UID: \"602654da-a6d1-40df-a05a-64f50a7879a0\") " Oct 30 03:45:58.418071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.418042 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume" pod "602654da-a6d1-40df-a05a-64f50a7879a0" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:58.418653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.418627 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume" pod "602654da-a6d1-40df-a05a-64f50a7879a0" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:58.420628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.420604 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq" pod "602654da-a6d1-40df-a05a-64f50a7879a0" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0"). InnerVolumeSpecName "kube-api-access-ksxjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:58.420831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.420804 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials" pod "602654da-a6d1-40df-a05a-64f50a7879a0" (UID: "602654da-a6d1-40df-a05a-64f50a7879a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:58.519176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.519144 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxjq\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-kube-api-access-ksxjq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.519176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.519171 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.519372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.519185 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602654da-a6d1-40df-a05a-64f50a7879a0-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.519372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.519197 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602654da-a6d1-40df-a05a-64f50a7879a0-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.625566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.625473 4560 generic.go:356] "Generic (PLEG): container finished" podID="602654da-a6d1-40df-a05a-64f50a7879a0" containerID="a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d" exitCode=0 Oct 30 03:45:58.625566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.625513 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" Oct 30 03:45:58.625566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.625558 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" event={"ID":"602654da-a6d1-40df-a05a-64f50a7879a0","Type":"ContainerDied","Data":"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d"} Oct 30 03:45:58.625808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.625614 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm" event={"ID":"602654da-a6d1-40df-a05a-64f50a7879a0","Type":"ContainerDied","Data":"0bccd7390080aac07865c9fc067fb126781786947b447155d3f91e836e0d4e2e"} Oct 30 03:45:58.625808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.625641 4560 scope.go:122] "RemoveContainer" containerID="a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d" Oct 30 03:45:58.635894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.635875 4560 scope.go:122] "RemoveContainer" containerID="a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d" Oct 30 03:45:58.636188 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:45:58.636168 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d\": container with ID starting with a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d not found: ID does not exist" containerID="a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d" Oct 30 03:45:58.636242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.636196 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d"} err="failed to get container status \"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d\": rpc error: code = NotFound desc = could not find container \"a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d\": container with ID starting with a0db0f692379860a135ab31b448230b5f0df171b8671eff4bb265a6fd8d7eb7d not found: ID does not exist" Oct 30 03:45:58.677415 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.677383 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:45:58.689904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:45:58.689876 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-566fhztbm"] Oct 30 03:46:00.154124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:00.154093 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602654da-a6d1-40df-a05a-64f50a7879a0" path="/var/lib/kubelet/pods/602654da-a6d1-40df-a05a-64f50a7879a0/volumes" Oct 30 03:46:09.008766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.008732 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:46:09.009363 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009349 4560 cpu_manager.go:414] "containerMap: removing container" podUID="602654da-a6d1-40df-a05a-64f50a7879a0" containerName="payload-processing" Oct 30 03:46:09.009410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009364 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602654da-a6d1-40df-a05a-64f50a7879a0" containerName="payload-processing" Oct 30 03:46:09.009410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009403 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:46:09.009410 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009409 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2e4f94-9168-4b5a-ac63-9fee6d114352" containerName="cleanup" Oct 30 03:46:09.009546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009500 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="602654da-a6d1-40df-a05a-64f50a7879a0" containerName="payload-processing" Oct 30 03:46:09.009546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009510 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b0d9-1861-442c-bda4-b5ccfb35cc24" containerName="cleanup" Oct 30 03:46:09.009546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009535 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c08a1c-aa8b-4d6e-98cd-21fd050a50d1" containerName="cleanup" Oct 30 03:46:09.009546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.009543 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88539de-8465-48a9-8d8c-4840f361e623" containerName="cleanup" Oct 30 03:46:09.014954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.014930 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.033083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.033037 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042084 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7a0ff47fb860\"" Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042173 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042179 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042211 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnpl6\"" Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042782 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:46:09.042838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.042836 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:46:09.169440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.169404 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkkg\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.169630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.169571 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.169630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.169620 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.169787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.169762 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.271445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.271357 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.271445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.271441 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.271703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.271552 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.271703 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.271607 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkkg\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.272175 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.272141 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.272281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.272192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.274019 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.273993 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.291223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.291200 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkkg\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\") pod \"payload-processing-external-model-7b6f7fb4f5-z4gp4\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.329717 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.329691 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:09.631997 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.631964 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:46:09.633103 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:46:09.633078 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61620dbb_c75f_4bb9_a428_73b99e929306.slice/crio-9929b874edf05daae4ddd9e665ac190300daceca247d46a6a91d342d145714fc WatchSource:0}: Error finding container 9929b874edf05daae4ddd9e665ac190300daceca247d46a6a91d342d145714fc: Status 404 returned error can't find the container with id 9929b874edf05daae4ddd9e665ac190300daceca247d46a6a91d342d145714fc Oct 30 03:46:09.693595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:09.693558 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" event={"ID":"61620dbb-c75f-4bb9-a428-73b99e929306","Type":"ContainerStarted","Data":"9929b874edf05daae4ddd9e665ac190300daceca247d46a6a91d342d145714fc"} Oct 30 03:46:10.701491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:10.701453 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" event={"ID":"61620dbb-c75f-4bb9-a428-73b99e929306","Type":"ContainerStarted","Data":"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33"} Oct 30 03:46:10.701911 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:10.701557 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:10.742070 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:10.742018 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" podStartSLOduration=2.74200318 podStartE2EDuration="2.74200318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:10.740677345 +0000 UTC m=+2598.725797233" watchObservedRunningTime="2026-10-30 03:46:10.74200318 +0000 UTC m=+2598.727123068" Oct 30 03:46:11.749461 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:46:11.749413 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:46:21.712424 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:21.712391 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:46:41.750768 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:46:41.750733 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:46:50.624801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.624751 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:46:50.631530 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.631497 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.644660 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.644628 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:46:50.653697 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.653664 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f0eb942fa2d9\"" Oct 30 03:46:50.786030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.785991 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.786331 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.786127 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.786331 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.786196 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2mg\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.786535 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.786348 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887112 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887020 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887112 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887088 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2mg\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887850 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.887965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.887952 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.889767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.889742 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.907420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.907390 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2mg\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\") pod \"payload-processing-external-model-d5bf6f9f-64gcz\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:50.947388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:50.947353 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:51.250113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:51.250086 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:46:51.250744 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:46:51.250716 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999986d6_42dc_4ff3_b54e_0e8556343a46.slice/crio-d22ededd54810a527974ef175aafa4a72df4a17e0b1314f42c79390b04ad120b WatchSource:0}: Error finding container d22ededd54810a527974ef175aafa4a72df4a17e0b1314f42c79390b04ad120b: Status 404 returned error can't find the container with id d22ededd54810a527974ef175aafa4a72df4a17e0b1314f42c79390b04ad120b Oct 30 03:46:51.960240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:51.960199 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" event={"ID":"999986d6-42dc-4ff3-b54e-0e8556343a46","Type":"ContainerStarted","Data":"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d"} Oct 30 03:46:51.960726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:51.960263 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" event={"ID":"999986d6-42dc-4ff3-b54e-0e8556343a46","Type":"ContainerStarted","Data":"d22ededd54810a527974ef175aafa4a72df4a17e0b1314f42c79390b04ad120b"} Oct 30 03:46:51.960726 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:51.960374 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:46:52.000326 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:52.000276 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" podStartSLOduration=2.000260963 podStartE2EDuration="2.000260963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:51.998554161 +0000 UTC m=+2639.983674049" watchObservedRunningTime="2026-10-30 03:46:52.000260963 +0000 UTC m=+2639.985380850" Oct 30 03:46:53.222324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.222286 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg"] Oct 30 03:46:53.228183 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.228159 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.243158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.243134 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fqfn6\"" Oct 30 03:46:53.260495 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.260419 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg"] Oct 30 03:46:53.314624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.314585 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.314825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.314691 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.314905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.314843 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.314905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.314881 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.315044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.315022 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.315176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.315157 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.315241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.315201 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.315349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.315332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvxb8\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-kube-api-access-tvxb8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.315416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.315367 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417038 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.416999 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417057 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417101 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvxb8\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-kube-api-access-tvxb8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417120 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417154 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417211 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417200 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417281 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417476 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417547 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417489 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417581 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417589 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.417794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.417644 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.418132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.418106 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.419655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.419636 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.419753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.419639 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.437145 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.437120 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.437549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.437497 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvxb8\" (UniqueName: \"kubernetes.io/projected/bdecf353-c6f0-4457-bdd4-98a4722c6ca9-kube-api-access-tvxb8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zkqzg\" (UID: \"bdecf353-c6f0-4457-bdd4-98a4722c6ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.543171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.543093 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:53.869808 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.869779 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg"] Oct 30 03:46:53.871317 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:46:53.871291 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdecf353_c6f0_4457_bdd4_98a4722c6ca9.slice/crio-a25805dd47a9dba4bc2c1994cc43d620bd223c27a32651c75cfac326f845d0b0 WatchSource:0}: Error finding container a25805dd47a9dba4bc2c1994cc43d620bd223c27a32651c75cfac326f845d0b0: Status 404 returned error can't find the container with id a25805dd47a9dba4bc2c1994cc43d620bd223c27a32651c75cfac326f845d0b0 Oct 30 03:46:53.874185 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.874153 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:46:53.874278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.874217 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:46:53.874278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.874243 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:46:53.976468 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.976434 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" event={"ID":"bdecf353-c6f0-4457-bdd4-98a4722c6ca9","Type":"ContainerStarted","Data":"035b5cb5302f9b6155b375ada9ec24c3883ad6707c09b8162d73cbe75f07b309"} Oct 30 03:46:53.976643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:53.976491 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" event={"ID":"bdecf353-c6f0-4457-bdd4-98a4722c6ca9","Type":"ContainerStarted","Data":"a25805dd47a9dba4bc2c1994cc43d620bd223c27a32651c75cfac326f845d0b0"} Oct 30 03:46:54.019159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:54.019100 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podStartSLOduration=1.019081886 podStartE2EDuration="1.019081886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:54.018537492 +0000 UTC m=+2642.003657380" watchObservedRunningTime="2026-10-30 03:46:54.019081886 +0000 UTC m=+2642.004201777" Oct 30 03:46:54.543986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:54.543938 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:46:54.546439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:54.546391 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:54.546628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:54.546467 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:46:55.544106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:55.544065 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:55.544528 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:55.544120 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:46:56.543861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:56.543821 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:56.544064 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:56.543880 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:46:57.543864 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:57.543829 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:57.544247 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:57.543887 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:46:58.543573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:58.543510 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:58.543763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:58.543596 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:46:59.543628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:59.543589 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:46:59.544037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:46:59.543647 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:47:00.543880 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:00.543830 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:47:00.544349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:00.543901 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:47:01.543708 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:01.543665 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 03:47:01.543892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:01.543745 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 03:47:01.708846 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:01.708805 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:47:01.712130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:01.712066 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:47:02.550046 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:02.550013 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:47:02.550440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:02.550420 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:47:02.551266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:02.551241 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" Oct 30 03:47:06.709647 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:06.709546 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" podUID="999986d6-42dc-4ff3-b54e-0e8556343a46" containerName="payload-processing" containerID="cri-o://7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d" gracePeriod=25 Oct 30 03:47:06.976941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:06.976883 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:47:07.010183 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.010155 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:07.010883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.010867 4560 cpu_manager.go:414] "containerMap: removing container" podUID="999986d6-42dc-4ff3-b54e-0e8556343a46" containerName="payload-processing" Oct 30 03:47:07.010936 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.010885 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999986d6-42dc-4ff3-b54e-0e8556343a46" containerName="payload-processing" Oct 30 03:47:07.011047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.011037 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="999986d6-42dc-4ff3-b54e-0e8556343a46" containerName="payload-processing" Oct 30 03:47:07.014483 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.014466 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.032015 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.031981 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"openshift-service-ca.crt\"" Oct 30 03:47:07.033327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.033306 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-5cfdc2e397bd\"" Oct 30 03:47:07.033641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.033621 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-e2e--4f8d4144-dockercfg-wmn86\"" Oct 30 03:47:07.033882 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.033865 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:47:07.034161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.034144 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"kube-root-ca.crt\"" Oct 30 03:47:07.034398 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.034382 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"routing-overlay\"" Oct 30 03:47:07.038867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.038837 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\") pod \"999986d6-42dc-4ff3-b54e-0e8556343a46\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " Oct 30 03:47:07.038979 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.038921 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\") pod \"999986d6-42dc-4ff3-b54e-0e8556343a46\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " Oct 30 03:47:07.039043 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.039028 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\") pod \"999986d6-42dc-4ff3-b54e-0e8556343a46\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " Oct 30 03:47:07.039298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.039093 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\") pod \"999986d6-42dc-4ff3-b54e-0e8556343a46\" (UID: \"999986d6-42dc-4ff3-b54e-0e8556343a46\") " Oct 30 03:47:07.040098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.040073 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume" pod "999986d6-42dc-4ff3-b54e-0e8556343a46" (UID: "999986d6-42dc-4ff3-b54e-0e8556343a46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:07.044616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.042892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume" pod "999986d6-42dc-4ff3-b54e-0e8556343a46" (UID: "999986d6-42dc-4ff3-b54e-0e8556343a46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:07.044616 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.043547 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg" pod "999986d6-42dc-4ff3-b54e-0e8556343a46" (UID: "999986d6-42dc-4ff3-b54e-0e8556343a46"). InnerVolumeSpecName "kube-api-access-7c2mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:07.047643 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.047607 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:07.048790 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.048768 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials" pod "999986d6-42dc-4ff3-b54e-0e8556343a46" (UID: "999986d6-42dc-4ff3-b54e-0e8556343a46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:07.063340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.063312 4560 generic.go:356] "Generic (PLEG): container finished" podID="999986d6-42dc-4ff3-b54e-0e8556343a46" containerID="7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d" exitCode=0 Oct 30 03:47:07.063456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.063365 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" Oct 30 03:47:07.063456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.063392 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" event={"ID":"999986d6-42dc-4ff3-b54e-0e8556343a46","Type":"ContainerDied","Data":"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d"} Oct 30 03:47:07.063456 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.063447 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz" event={"ID":"999986d6-42dc-4ff3-b54e-0e8556343a46","Type":"ContainerDied","Data":"d22ededd54810a527974ef175aafa4a72df4a17e0b1314f42c79390b04ad120b"} Oct 30 03:47:07.063624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.063476 4560 scope.go:122] "RemoveContainer" containerID="7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d" Oct 30 03:47:07.075209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.075184 4560 scope.go:122] "RemoveContainer" containerID="7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d" Oct 30 03:47:07.075550 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:07.075499 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d\": container with ID starting with 7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d not found: ID does not exist" containerID="7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d" Oct 30 03:47:07.075631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.075543 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d"} err="failed to get container status \"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d\": rpc error: code = NotFound desc = could not find container \"7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d\": container with ID starting with 7833970bd5772be660e52e59de04647ab31e5152cbc3263491af620610dc0e1d not found: ID does not exist" Oct 30 03:47:07.118898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.118863 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:47:07.121606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.121581 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-64gcz"] Oct 30 03:47:07.140830 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.140799 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.140962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.140881 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86k2s\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.141083 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141057 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.141190 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141162 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.141327 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141315 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:07.141397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141335 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999986d6-42dc-4ff3-b54e-0e8556343a46-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:07.141397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141351 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2mg\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-kube-api-access-7c2mg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:07.141397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.141368 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/999986d6-42dc-4ff3-b54e-0e8556343a46-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:07.242946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.242860 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k2s\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.243127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.243088 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.244005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.243342 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.244005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.243605 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.244230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.244015 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.244649 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.244631 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.246091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.246068 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.263294 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.263265 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86k2s\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.353333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.353288 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:07.658223 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:07.658183 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dffbc6a_1b9c_4947_a6e5_decb6b477106.slice/crio-c0cf4f9ddfbbc96d8cf77a4c885efe97354110eb30c7f5cd9bb539df602cc94c WatchSource:0}: Error finding container c0cf4f9ddfbbc96d8cf77a4c885efe97354110eb30c7f5cd9bb539df602cc94c: Status 404 returned error can't find the container with id c0cf4f9ddfbbc96d8cf77a4c885efe97354110eb30c7f5cd9bb539df602cc94c Oct 30 03:47:07.658443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:07.658419 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:08.071797 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.071753 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" event={"ID":"3dffbc6a-1b9c-4947-a6e5-decb6b477106","Type":"ContainerStarted","Data":"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a"} Oct 30 03:47:08.072277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.071815 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:08.072277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.071835 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" event={"ID":"3dffbc6a-1b9c-4947-a6e5-decb6b477106","Type":"ContainerStarted","Data":"c0cf4f9ddfbbc96d8cf77a4c885efe97354110eb30c7f5cd9bb539df602cc94c"} Oct 30 03:47:08.112371 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.112319 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" podStartSLOduration=2.112306293 podStartE2EDuration="2.112306293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:08.110696497 +0000 UTC m=+2656.095816386" watchObservedRunningTime="2026-10-30 03:47:08.112306293 +0000 UTC m=+2656.097426213" Oct 30 03:47:08.154447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.154411 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999986d6-42dc-4ff3-b54e-0e8556343a46" path="/var/lib/kubelet/pods/999986d6-42dc-4ff3-b54e-0e8556343a46/volumes" Oct 30 03:47:08.279588 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.279544 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n"] Oct 30 03:47:08.284585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.284560 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.307340 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.307304 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n"] Oct 30 03:47:08.356443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356356 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356399 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356421 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2mm\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-kube-api-access-jb2mm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356613 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0806baf1-a58a-48df-8c59-aabafc920d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356655 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356775 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356803 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0806baf1-a58a-48df-8c59-aabafc920d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356842 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.356955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.356937 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458171 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458417 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458230 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458417 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458266 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458417 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458417 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458300 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2mm\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-kube-api-access-jb2mm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458452 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0806baf1-a58a-48df-8c59-aabafc920d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458548 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458707 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458737 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458757 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0806baf1-a58a-48df-8c59-aabafc920d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458712 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.458810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458801 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.459031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.458963 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.459358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.459339 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0806baf1-a58a-48df-8c59-aabafc920d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.460607 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.460578 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0806baf1-a58a-48df-8c59-aabafc920d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.460932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.460913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0806baf1-a58a-48df-8c59-aabafc920d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.477682 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.477654 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.477879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.477860 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2mm\" (UniqueName: \"kubernetes.io/projected/0806baf1-a58a-48df-8c59-aabafc920d93-kube-api-access-jb2mm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88m6n\" (UID: \"0806baf1-a58a-48df-8c59-aabafc920d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.604317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.604288 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:08.924747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.924717 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n"] Oct 30 03:47:08.925358 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:08.925329 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0806baf1_a58a_48df_8c59_aabafc920d93.slice/crio-9024d8f3be417f846ed6b1413a4ba4c1ba7fae7297a3a32b1d0762ca1d0a6f7d WatchSource:0}: Error finding container 9024d8f3be417f846ed6b1413a4ba4c1ba7fae7297a3a32b1d0762ca1d0a6f7d: Status 404 returned error can't find the container with id 9024d8f3be417f846ed6b1413a4ba4c1ba7fae7297a3a32b1d0762ca1d0a6f7d Oct 30 03:47:08.928277 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.928241 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:08.928394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.928325 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:08.928394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:08.928372 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:09.080484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.080438 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" event={"ID":"0806baf1-a58a-48df-8c59-aabafc920d93","Type":"ContainerStarted","Data":"89e150edeafc480a4ad6618e25e019be12631514f485d853d80a74dda640af7a"} Oct 30 03:47:09.080890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.080529 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" event={"ID":"0806baf1-a58a-48df-8c59-aabafc920d93","Type":"ContainerStarted","Data":"9024d8f3be417f846ed6b1413a4ba4c1ba7fae7297a3a32b1d0762ca1d0a6f7d"} Oct 30 03:47:09.123249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.123186 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podStartSLOduration=1.123165758 podStartE2EDuration="1.123165758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:09.120701266 +0000 UTC m=+2657.105821179" watchObservedRunningTime="2026-10-30 03:47:09.123165758 +0000 UTC m=+2657.108285648" Oct 30 03:47:09.605094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.605062 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:09.606792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.606766 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:09.606902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:09.606839 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:10.605480 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:10.605445 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:10.605916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:10.605500 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:11.605233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:11.605196 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:11.605435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:11.605263 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:11.803537 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:11.801233 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:47:12.604835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:12.604795 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:12.605011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:12.604855 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:13.604963 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:13.604930 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:13.605353 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:13.604982 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:14.033567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.033532 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:47:14.038236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.038215 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.078669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.078632 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-eda87f61d8ec\"" Oct 30 03:47:14.079262 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.079234 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:47:14.132472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.132436 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.132657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.132484 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7vm\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.132657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.132640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.132741 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.132696 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.233492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.233454 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.233492 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.233494 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7vm\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.233791 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.233558 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.233791 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.233621 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.234263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.234239 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.234376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.234356 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.236128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.236102 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.277215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.277183 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7vm\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.378450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.378347 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:47:14.378906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.378879 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:14.605743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.605710 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:14.606120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.605763 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:14.696541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:14.696496 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:47:14.697579 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:14.697551 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc79c25c_0500_4625_93fb_1fdbf8b3c031.slice/crio-3d14b594eb78ff26ffb334a5d99a4cb7ff5bfcc6a803740d46a160209b504f02 WatchSource:0}: Error finding container 3d14b594eb78ff26ffb334a5d99a4cb7ff5bfcc6a803740d46a160209b504f02: Status 404 returned error can't find the container with id 3d14b594eb78ff26ffb334a5d99a4cb7ff5bfcc6a803740d46a160209b504f02 Oct 30 03:47:15.128060 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:15.127931 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" event={"ID":"bc79c25c-0500-4625-93fb-1fdbf8b3c031","Type":"ContainerStarted","Data":"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7"} Oct 30 03:47:15.128233 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:15.128108 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" event={"ID":"bc79c25c-0500-4625-93fb-1fdbf8b3c031","Type":"ContainerStarted","Data":"3d14b594eb78ff26ffb334a5d99a4cb7ff5bfcc6a803740d46a160209b504f02"} Oct 30 03:47:15.170905 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:15.170850 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" podStartSLOduration=1.170835543 podStartE2EDuration="1.170835543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:15.167529674 +0000 UTC m=+2663.152649553" watchObservedRunningTime="2026-10-30 03:47:15.170835543 +0000 UTC m=+2663.155955456" Oct 30 03:47:15.605118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:15.605078 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:15.605298 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:15.605143 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:16.134723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:16.134688 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:16.605441 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:16.605395 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 03:47:16.605660 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:16.605480 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 03:47:17.730120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:17.730084 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:17.730613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:17.730585 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:17.731454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:17.731434 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" Oct 30 03:47:18.551699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:18.551654 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:47:18.551935 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:18.551728 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:47:19.084316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.084280 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:19.379136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.379008 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" podUID="61620dbb-c75f-4bb9-a428-73b99e929306" containerName="payload-processing" containerID="cri-o://13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33" gracePeriod=25 Oct 30 03:47:19.702772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.702742 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:47:19.814688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.814636 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\") pod \"61620dbb-c75f-4bb9-a428-73b99e929306\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " Oct 30 03:47:19.814890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.814731 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\") pod \"61620dbb-c75f-4bb9-a428-73b99e929306\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " Oct 30 03:47:19.814890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.814800 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\") pod \"61620dbb-c75f-4bb9-a428-73b99e929306\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " Oct 30 03:47:19.814890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.814854 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\") pod \"61620dbb-c75f-4bb9-a428-73b99e929306\" (UID: \"61620dbb-c75f-4bb9-a428-73b99e929306\") " Oct 30 03:47:19.815781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.815754 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume" pod "61620dbb-c75f-4bb9-a428-73b99e929306" (UID: "61620dbb-c75f-4bb9-a428-73b99e929306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:19.816100 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.816078 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume" pod "61620dbb-c75f-4bb9-a428-73b99e929306" (UID: "61620dbb-c75f-4bb9-a428-73b99e929306"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:19.817201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.817178 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg" pod "61620dbb-c75f-4bb9-a428-73b99e929306" (UID: "61620dbb-c75f-4bb9-a428-73b99e929306"). InnerVolumeSpecName "kube-api-access-hzkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:19.817367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.817342 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials" pod "61620dbb-c75f-4bb9-a428-73b99e929306" (UID: "61620dbb-c75f-4bb9-a428-73b99e929306"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:19.917433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.917340 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkkg\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-kube-api-access-hzkkg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:19.917433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.917388 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:19.917433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.917404 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61620dbb-c75f-4bb9-a428-73b99e929306-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:19.917433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:19.917421 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61620dbb-c75f-4bb9-a428-73b99e929306-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:20.187001 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.186914 4560 generic.go:356] "Generic (PLEG): container finished" podID="61620dbb-c75f-4bb9-a428-73b99e929306" containerID="13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33" exitCode=0 Oct 30 03:47:20.188031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.188006 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" Oct 30 03:47:20.188165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.188006 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" event={"ID":"61620dbb-c75f-4bb9-a428-73b99e929306","Type":"ContainerDied","Data":"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33"} Oct 30 03:47:20.188227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.188181 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4" event={"ID":"61620dbb-c75f-4bb9-a428-73b99e929306","Type":"ContainerDied","Data":"9929b874edf05daae4ddd9e665ac190300daceca247d46a6a91d342d145714fc"} Oct 30 03:47:20.188274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.188222 4560 scope.go:122] "RemoveContainer" containerID="13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33" Oct 30 03:47:20.203124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.203104 4560 scope.go:122] "RemoveContainer" containerID="13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33" Oct 30 03:47:20.203471 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:20.203444 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33\": container with ID starting with 13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33 not found: ID does not exist" containerID="13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33" Oct 30 03:47:20.203617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.203479 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33"} err="failed to get container status \"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33\": rpc error: code = NotFound desc = could not find container \"13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33\": container with ID starting with 13d37f75719d16a1ce3e677095d53444fe9eb24d22f1af4a968d26bf98270f33 not found: ID does not exist" Oct 30 03:47:20.240130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.240103 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:47:20.252272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:20.252246 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6f7fb4f5-z4gp4"] Oct 30 03:47:22.154952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:22.154918 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61620dbb-c75f-4bb9-a428-73b99e929306" path="/var/lib/kubelet/pods/61620dbb-c75f-4bb9-a428-73b99e929306/volumes" Oct 30 03:47:27.146631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:27.146601 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:47:27.216848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:27.216813 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:32.217957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.217871 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" podUID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" containerName="payload-processing" containerID="cri-o://7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a" gracePeriod=25 Oct 30 03:47:32.500863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.500833 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:32.571884 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.571848 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\") pod \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " Oct 30 03:47:32.572085 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.572046 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\") pod \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " Oct 30 03:47:32.572153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.572082 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\") pod \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " Oct 30 03:47:32.572260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.572243 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\") pod \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\" (UID: \"3dffbc6a-1b9c-4947-a6e5-decb6b477106\") " Oct 30 03:47:32.572724 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.572619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume" pod "3dffbc6a-1b9c-4947-a6e5-decb6b477106" (UID: "3dffbc6a-1b9c-4947-a6e5-decb6b477106"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:32.572932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.572903 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume" pod "3dffbc6a-1b9c-4947-a6e5-decb6b477106" (UID: "3dffbc6a-1b9c-4947-a6e5-decb6b477106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:32.575463 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.575426 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials" pod "3dffbc6a-1b9c-4947-a6e5-decb6b477106" (UID: "3dffbc6a-1b9c-4947-a6e5-decb6b477106"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:32.594632 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.594602 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s" pod "3dffbc6a-1b9c-4947-a6e5-decb6b477106" (UID: "3dffbc6a-1b9c-4947-a6e5-decb6b477106"). InnerVolumeSpecName "kube-api-access-86k2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:32.673806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.673770 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:32.673806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.673801 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:32.673806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.673811 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86k2s\" (UniqueName: \"kubernetes.io/projected/3dffbc6a-1b9c-4947-a6e5-decb6b477106-kube-api-access-86k2s\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:32.674047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:32.673821 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dffbc6a-1b9c-4947-a6e5-decb6b477106-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:33.276355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.276312 4560 generic.go:356] "Generic (PLEG): container finished" podID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" containerID="7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a" exitCode=0 Oct 30 03:47:33.276878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.276381 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" Oct 30 03:47:33.276878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.276392 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" event={"ID":"3dffbc6a-1b9c-4947-a6e5-decb6b477106","Type":"ContainerDied","Data":"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a"} Oct 30 03:47:33.276878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.276445 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v" event={"ID":"3dffbc6a-1b9c-4947-a6e5-decb6b477106","Type":"ContainerDied","Data":"c0cf4f9ddfbbc96d8cf77a4c885efe97354110eb30c7f5cd9bb539df602cc94c"} Oct 30 03:47:33.276878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.276471 4560 scope.go:122] "RemoveContainer" containerID="7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a" Oct 30 03:47:33.292421 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.292399 4560 scope.go:122] "RemoveContainer" containerID="7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a" Oct 30 03:47:33.292807 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:33.292782 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a\": container with ID starting with 7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a not found: ID does not exist" containerID="7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a" Oct 30 03:47:33.292890 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.292820 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a"} err="failed to get container status \"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a\": rpc error: code = NotFound desc = could not find container \"7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a\": container with ID starting with 7a126e0165445bea3f3e2fac6025a403bd36387bee258c8b94e58ef1319b987a not found: ID does not exist" Oct 30 03:47:33.330375 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.330333 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:33.333685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:33.333656 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dbvvq5v"] Oct 30 03:47:34.154761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.154727 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" path="/var/lib/kubelet/pods/3dffbc6a-1b9c-4947-a6e5-decb6b477106/volumes" Oct 30 03:47:34.957109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957072 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:34.957750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957733 4560 cpu_manager.go:414] "containerMap: removing container" podUID="61620dbb-c75f-4bb9-a428-73b99e929306" containerName="payload-processing" Oct 30 03:47:34.957874 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957751 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61620dbb-c75f-4bb9-a428-73b99e929306" containerName="payload-processing" Oct 30 03:47:34.957874 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957781 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" containerName="payload-processing" Oct 30 03:47:34.957874 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957788 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" containerName="payload-processing" Oct 30 03:47:34.957976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957889 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dffbc6a-1b9c-4947-a6e5-decb6b477106" containerName="payload-processing" Oct 30 03:47:34.957976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.957903 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="61620dbb-c75f-4bb9-a428-73b99e929306" containerName="payload-processing" Oct 30 03:47:34.962653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.962637 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:34.990737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.990706 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kcbcw\"" Oct 30 03:47:34.990915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.990830 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:47:34.990996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.990918 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:47:34.990996 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.990977 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:47:34.991121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.991046 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7e23b6c624f1\"" Oct 30 03:47:34.991180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.991123 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:47:34.994638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:34.994611 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:35.005354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.004279 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.005354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.004374 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.005354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.004608 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.005354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.005021 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtm7b\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.108591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.108554 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm7b\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.108794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.108653 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.108794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.108706 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.108794 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.108788 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.109709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.109674 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.109852 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.109761 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.111344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.111317 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.129381 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.129352 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm7b\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\") pod \"payload-processing-external-model-5bc757568-bkxtv\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.281899 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.281805 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:35.595974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:35.595947 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:35.597406 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:35.597375 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fc251a_57fe_4af4_b53c_a9c114176945.slice/crio-e54181afef0c6afe886cb1d74daab31b364fa5c580f1a92e2934c417a0304f7b WatchSource:0}: Error finding container e54181afef0c6afe886cb1d74daab31b364fa5c580f1a92e2934c417a0304f7b: Status 404 returned error can't find the container with id e54181afef0c6afe886cb1d74daab31b364fa5c580f1a92e2934c417a0304f7b Oct 30 03:47:36.302282 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:36.302242 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" event={"ID":"d5fc251a-57fe-4af4-b53c-a9c114176945","Type":"ContainerStarted","Data":"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca"} Oct 30 03:47:36.302732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:36.302308 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" event={"ID":"d5fc251a-57fe-4af4-b53c-a9c114176945","Type":"ContainerStarted","Data":"e54181afef0c6afe886cb1d74daab31b364fa5c580f1a92e2934c417a0304f7b"} Oct 30 03:47:36.302732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:36.302340 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:36.342992 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:36.342935 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" podStartSLOduration=2.342919862 podStartE2EDuration="2.342919862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:36.341357915 +0000 UTC m=+2684.326477795" watchObservedRunningTime="2026-10-30 03:47:36.342919862 +0000 UTC m=+2684.328039749" Oct 30 03:47:39.463137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:39.463098 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:41.761713 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:41.761669 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:47:43.692740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.692673 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:43.698105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.698081 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.716731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.716702 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:43.821029 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.820993 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.821204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.821046 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.821204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.821137 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjjc\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.821204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.821196 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.922798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.922759 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.922980 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.922929 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.923048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.922985 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.923048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.923022 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjjc\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.923728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.923700 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.923860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.923802 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.925472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.925431 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:43.942051 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:43.942020 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjjc\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\") pod \"payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:44.012832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.012800 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:44.322030 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:44.321985 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e91b0c_42e5_418f_9151_4a68a0bc4574.slice/crio-f19b3c782e09e7e7232e678b0081d7355cbd326edfd0c0d6d65d2ae80c98c4af WatchSource:0}: Error finding container f19b3c782e09e7e7232e678b0081d7355cbd326edfd0c0d6d65d2ae80c98c4af: Status 404 returned error can't find the container with id f19b3c782e09e7e7232e678b0081d7355cbd326edfd0c0d6d65d2ae80c98c4af Oct 30 03:47:44.322893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.322859 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:44.357577 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.357512 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" event={"ID":"a8e91b0c-42e5-418f-9151-4a68a0bc4574","Type":"ContainerStarted","Data":"f19b3c782e09e7e7232e678b0081d7355cbd326edfd0c0d6d65d2ae80c98c4af"} Oct 30 03:47:44.403909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.403872 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:44.407225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.407194 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:47:44.463980 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.463938 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" podUID="d5fc251a-57fe-4af4-b53c-a9c114176945" containerName="payload-processing" containerID="cri-o://dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca" gracePeriod=25 Oct 30 03:47:44.716384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.716358 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:44.833909 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.833813 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\") pod \"d5fc251a-57fe-4af4-b53c-a9c114176945\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " Oct 30 03:47:44.834095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.833971 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\") pod \"d5fc251a-57fe-4af4-b53c-a9c114176945\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " Oct 30 03:47:44.834095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834059 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\") pod \"d5fc251a-57fe-4af4-b53c-a9c114176945\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " Oct 30 03:47:44.834223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834104 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\") pod \"d5fc251a-57fe-4af4-b53c-a9c114176945\" (UID: \"d5fc251a-57fe-4af4-b53c-a9c114176945\") " Oct 30 03:47:44.834280 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834234 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume" pod "d5fc251a-57fe-4af4-b53c-a9c114176945" (UID: "d5fc251a-57fe-4af4-b53c-a9c114176945"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:44.834538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834499 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume" pod "d5fc251a-57fe-4af4-b53c-a9c114176945" (UID: "d5fc251a-57fe-4af4-b53c-a9c114176945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:44.834681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834614 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:44.834681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.834633 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5fc251a-57fe-4af4-b53c-a9c114176945-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:44.836236 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.836214 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b" pod "d5fc251a-57fe-4af4-b53c-a9c114176945" (UID: "d5fc251a-57fe-4af4-b53c-a9c114176945"). InnerVolumeSpecName "kube-api-access-mtm7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:44.836693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.836671 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials" pod "d5fc251a-57fe-4af4-b53c-a9c114176945" (UID: "d5fc251a-57fe-4af4-b53c-a9c114176945"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:44.935606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.935571 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtm7b\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-kube-api-access-mtm7b\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:44.935606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:44.935600 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5fc251a-57fe-4af4-b53c-a9c114176945-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:45.365175 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.365138 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" event={"ID":"a8e91b0c-42e5-418f-9151-4a68a0bc4574","Type":"ContainerStarted","Data":"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492"} Oct 30 03:47:45.365384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.365365 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:45.366544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.366498 4560 generic.go:356] "Generic (PLEG): container finished" podID="d5fc251a-57fe-4af4-b53c-a9c114176945" containerID="dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca" exitCode=0 Oct 30 03:47:45.366666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.366570 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" Oct 30 03:47:45.366666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.366575 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" event={"ID":"d5fc251a-57fe-4af4-b53c-a9c114176945","Type":"ContainerDied","Data":"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca"} Oct 30 03:47:45.366666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.366617 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-bkxtv" event={"ID":"d5fc251a-57fe-4af4-b53c-a9c114176945","Type":"ContainerDied","Data":"e54181afef0c6afe886cb1d74daab31b364fa5c580f1a92e2934c417a0304f7b"} Oct 30 03:47:45.366666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.366639 4560 scope.go:122] "RemoveContainer" containerID="dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca" Oct 30 03:47:45.376878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.376854 4560 scope.go:122] "RemoveContainer" containerID="dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca" Oct 30 03:47:45.377185 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:45.377160 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca\": container with ID starting with dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca not found: ID does not exist" containerID="dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca" Oct 30 03:47:45.377278 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.377190 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca"} err="failed to get container status \"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca\": rpc error: code = NotFound desc = could not find container \"dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca\": container with ID starting with dd0e44b5d882161983eee0a9fa7239b816c76559b630a75644938898ebf047ca not found: ID does not exist" Oct 30 03:47:45.404147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.404098 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" podStartSLOduration=2.404086684 podStartE2EDuration="2.404086684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:45.403047985 +0000 UTC m=+2693.388167897" watchObservedRunningTime="2026-10-30 03:47:45.404086684 +0000 UTC m=+2693.389206572" Oct 30 03:47:45.440287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.440249 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:45.443065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:45.443038 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-bkxtv"] Oct 30 03:47:46.154698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:46.154665 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fc251a-57fe-4af4-b53c-a9c114176945" path="/var/lib/kubelet/pods/d5fc251a-57fe-4af4-b53c-a9c114176945/volumes" Oct 30 03:47:49.407974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.407676 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" podUID="603eb620-b75e-4c84-b172-96d94900f090" containerName="payload-processing" containerID="cri-o://54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7" gracePeriod=25 Oct 30 03:47:49.728037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.728003 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:47:49.810753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.810254 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\") pod \"603eb620-b75e-4c84-b172-96d94900f090\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " Oct 30 03:47:49.810753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.810331 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\") pod \"603eb620-b75e-4c84-b172-96d94900f090\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " Oct 30 03:47:49.810753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.810402 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\") pod \"603eb620-b75e-4c84-b172-96d94900f090\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " Oct 30 03:47:49.810753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.810545 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\") pod \"603eb620-b75e-4c84-b172-96d94900f090\" (UID: \"603eb620-b75e-4c84-b172-96d94900f090\") " Oct 30 03:47:49.811923 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.811412 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume" pod "603eb620-b75e-4c84-b172-96d94900f090" (UID: "603eb620-b75e-4c84-b172-96d94900f090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:49.813592 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.812238 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume" pod "603eb620-b75e-4c84-b172-96d94900f090" (UID: "603eb620-b75e-4c84-b172-96d94900f090"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:49.814737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.814707 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials" pod "603eb620-b75e-4c84-b172-96d94900f090" (UID: "603eb620-b75e-4c84-b172-96d94900f090"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:49.818008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.817979 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp" pod "603eb620-b75e-4c84-b172-96d94900f090" (UID: "603eb620-b75e-4c84-b172-96d94900f090"). InnerVolumeSpecName "kube-api-access-5jrcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:49.912254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.912155 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:49.912254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.912196 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:49.912254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.912213 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrcp\" (UniqueName: \"kubernetes.io/projected/603eb620-b75e-4c84-b172-96d94900f090-kube-api-access-5jrcp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:49.912254 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:49.912227 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603eb620-b75e-4c84-b172-96d94900f090-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:50.367627 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.367576 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" podUID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" containerName="payload-processing" containerID="cri-o://6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492" gracePeriod=25 Oct 30 03:47:50.418187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.417780 4560 generic.go:356] "Generic (PLEG): container finished" podID="603eb620-b75e-4c84-b172-96d94900f090" containerID="54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7" exitCode=0 Oct 30 03:47:50.418187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.417868 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" Oct 30 03:47:50.418187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.417897 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" event={"ID":"603eb620-b75e-4c84-b172-96d94900f090","Type":"ContainerDied","Data":"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7"} Oct 30 03:47:50.418187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.417952 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc" event={"ID":"603eb620-b75e-4c84-b172-96d94900f090","Type":"ContainerDied","Data":"ea7a3245162b27287f3833c89f4d06e5153437c8c8e351abd3c39ac36745e55d"} Oct 30 03:47:50.418187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.417986 4560 scope.go:122] "RemoveContainer" containerID="54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7" Oct 30 03:47:50.517565 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.514676 4560 scope.go:122] "RemoveContainer" containerID="54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7" Oct 30 03:47:50.528436 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:50.528142 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7\": container with ID starting with 54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7 not found: ID does not exist" containerID="54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7" Oct 30 03:47:50.528436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.528192 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7"} err="failed to get container status \"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7\": rpc error: code = NotFound desc = could not find container \"54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7\": container with ID starting with 54bde75f599f3a0d5ef6bdfa746d0ea4ef1448da2eebc8bddaf62f142bfc8bc7 not found: ID does not exist" Oct 30 03:47:50.536534 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.535049 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:47:50.545903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.545876 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-8565697sc"] Oct 30 03:47:50.950930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:50.950891 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:51.039802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.039762 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\") pod \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " Oct 30 03:47:51.039988 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.039896 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\") pod \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " Oct 30 03:47:51.039988 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.039958 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\") pod \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " Oct 30 03:47:51.040140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.040067 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\") pod \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\" (UID: \"a8e91b0c-42e5-418f-9151-4a68a0bc4574\") " Oct 30 03:47:51.040710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.040676 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume" pod "a8e91b0c-42e5-418f-9151-4a68a0bc4574" (UID: "a8e91b0c-42e5-418f-9151-4a68a0bc4574"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:51.041042 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.041021 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume" pod "a8e91b0c-42e5-418f-9151-4a68a0bc4574" (UID: "a8e91b0c-42e5-418f-9151-4a68a0bc4574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:51.042962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.042935 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials" pod "a8e91b0c-42e5-418f-9151-4a68a0bc4574" (UID: "a8e91b0c-42e5-418f-9151-4a68a0bc4574"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:51.043047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.042994 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc" pod "a8e91b0c-42e5-418f-9151-4a68a0bc4574" (UID: "a8e91b0c-42e5-418f-9151-4a68a0bc4574"). InnerVolumeSpecName "kube-api-access-ssjjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:51.142139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.142030 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjjc\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-kube-api-access-ssjjc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:51.142139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.142074 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:51.142139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.142088 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e91b0c-42e5-418f-9151-4a68a0bc4574-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:51.142139 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.142100 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8e91b0c-42e5-418f-9151-4a68a0bc4574-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:47:51.426434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.426347 4560 generic.go:356] "Generic (PLEG): container finished" podID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" containerID="6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492" exitCode=0 Oct 30 03:47:51.426434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.426395 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" Oct 30 03:47:51.426955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.426434 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" event={"ID":"a8e91b0c-42e5-418f-9151-4a68a0bc4574","Type":"ContainerDied","Data":"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492"} Oct 30 03:47:51.426955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.426486 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs" event={"ID":"a8e91b0c-42e5-418f-9151-4a68a0bc4574","Type":"ContainerDied","Data":"f19b3c782e09e7e7232e678b0081d7355cbd326edfd0c0d6d65d2ae80c98c4af"} Oct 30 03:47:51.426955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.426510 4560 scope.go:122] "RemoveContainer" containerID="6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492" Oct 30 03:47:51.436804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.436787 4560 scope.go:122] "RemoveContainer" containerID="6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492" Oct 30 03:47:51.437067 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:47:51.437049 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492\": container with ID starting with 6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492 not found: ID does not exist" containerID="6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492" Oct 30 03:47:51.437126 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.437075 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492"} err="failed to get container status \"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492\": rpc error: code = NotFound desc = could not find container \"6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492\": container with ID starting with 6fbd68a6cb9f591aa323fc72b2bb263059b6d177fe014dcb455b85b4d225d492 not found: ID does not exist" Oct 30 03:47:51.491003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.490961 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:51.503219 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:51.503168 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-398c25/payload-processing-external-model-e2e-extmod-a-398c25-6468n7dhs"] Oct 30 03:47:52.153573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:52.153541 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603eb620-b75e-4c84-b172-96d94900f090" path="/var/lib/kubelet/pods/603eb620-b75e-4c84-b172-96d94900f090/volumes" Oct 30 03:47:52.153962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:52.153949 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" path="/var/lib/kubelet/pods/a8e91b0c-42e5-418f-9151-4a68a0bc4574/volumes" Oct 30 03:47:53.390754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.390679 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75"] Oct 30 03:47:53.391602 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391583 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" containerName="payload-processing" Oct 30 03:47:53.391669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391603 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" containerName="payload-processing" Oct 30 03:47:53.391669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391622 4560 cpu_manager.go:414] "containerMap: removing container" podUID="603eb620-b75e-4c84-b172-96d94900f090" containerName="payload-processing" Oct 30 03:47:53.391669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391627 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603eb620-b75e-4c84-b172-96d94900f090" containerName="payload-processing" Oct 30 03:47:53.391669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391661 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d5fc251a-57fe-4af4-b53c-a9c114176945" containerName="payload-processing" Oct 30 03:47:53.391669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391667 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fc251a-57fe-4af4-b53c-a9c114176945" containerName="payload-processing" Oct 30 03:47:53.391819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391761 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fc251a-57fe-4af4-b53c-a9c114176945" containerName="payload-processing" Oct 30 03:47:53.391819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391771 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e91b0c-42e5-418f-9151-4a68a0bc4574" containerName="payload-processing" Oct 30 03:47:53.391819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.391779 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="603eb620-b75e-4c84-b172-96d94900f090" containerName="payload-processing" Oct 30 03:47:53.396729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.396706 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.421468 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.421425 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75"] Oct 30 03:47:53.469164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469127 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469171 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf485c0-995a-4d10-ac7f-dd565faf112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469284 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469325 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469367 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75k4l\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-kube-api-access-75k4l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469474 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469442 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469511 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469475 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469582 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469554 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.469635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.469621 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570593 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570619 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf485c0-995a-4d10-ac7f-dd565faf112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570678 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570741 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570790 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75k4l\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-kube-api-access-75k4l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570811 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.570872 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570839 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571075 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.570890 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.571067 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571258 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.571234 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.571290 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571333 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.571322 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.571626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.571608 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf485c0-995a-4d10-ac7f-dd565faf112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.573321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.573302 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.573617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.573597 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.589767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.589732 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.589920 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.589883 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75k4l\" (UniqueName: \"kubernetes.io/projected/dcf485c0-995a-4d10-ac7f-dd565faf112b-kube-api-access-75k4l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lkn75\" (UID: \"dcf485c0-995a-4d10-ac7f-dd565faf112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:53.710596 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:53.710564 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:54.038673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.038646 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75"] Oct 30 03:47:54.039852 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:54.039821 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf485c0_995a_4d10_ac7f_dd565faf112b.slice/crio-a39eeb60a107a66623654ab0accc7c0db9ff1c8cb1776c46a75ae6589ef17d0d WatchSource:0}: Error finding container a39eeb60a107a66623654ab0accc7c0db9ff1c8cb1776c46a75ae6589ef17d0d: Status 404 returned error can't find the container with id a39eeb60a107a66623654ab0accc7c0db9ff1c8cb1776c46a75ae6589ef17d0d Oct 30 03:47:54.042696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.042665 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:54.042753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.042734 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:54.042789 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.042761 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:47:54.449037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.448950 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" event={"ID":"dcf485c0-995a-4d10-ac7f-dd565faf112b","Type":"ContainerStarted","Data":"7292e176e9dee0af2eb4dbedcd06fea85f4adfa3cc81448f8dd38fcdace02b2c"} Oct 30 03:47:54.449037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.449001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" event={"ID":"dcf485c0-995a-4d10-ac7f-dd565faf112b","Type":"ContainerStarted","Data":"a39eeb60a107a66623654ab0accc7c0db9ff1c8cb1776c46a75ae6589ef17d0d"} Oct 30 03:47:54.497010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.496954 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podStartSLOduration=1.496940889 podStartE2EDuration="1.496940889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:54.492277811 +0000 UTC m=+2702.477397700" watchObservedRunningTime="2026-10-30 03:47:54.496940889 +0000 UTC m=+2702.482060851" Oct 30 03:47:54.711698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:54.711665 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:47:55.712738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:55.712699 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:47:55.713138 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:55.712761 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:47:56.581475 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.581433 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:47:56.585865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.585842 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.604435 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.604404 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:47:56.610234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.610201 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chst\" (UniqueName: \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.610413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.610273 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.613419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.613389 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ae0e75-dockercfg-2h7wl\"" Oct 30 03:47:56.613612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.613415 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ae0e75\"" Oct 30 03:47:56.693079 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.693035 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:47:56.698828 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.698794 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.709680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.705541 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:47:56.711541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.711385 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chst\" (UniqueName: \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.711541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.711490 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.713191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.713159 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:47:56.713626 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.713215 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:47:56.714681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.714617 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.738243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.738209 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chst\" (UniqueName: \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\") pod \"payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.812363 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.812327 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxng6\" (UniqueName: \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.812556 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.812381 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.901473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.901380 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:56.914133 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.914104 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxng6\" (UniqueName: \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.914271 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.914158 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.914936 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.914913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:56.933631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:56.933602 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxng6\" (UniqueName: \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\") pod \"payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:57.028924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.028681 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:57.226749 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.226723 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:47:57.227565 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:57.227540 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0326dcb_0dcc_49f7_a46f_3a1611e438bd.slice/crio-f31eaf1371fc0b29718c17ebaf6160ec403593c8cbf82b1abb1ee46c5ea0d820 WatchSource:0}: Error finding container f31eaf1371fc0b29718c17ebaf6160ec403593c8cbf82b1abb1ee46c5ea0d820: Status 404 returned error can't find the container with id f31eaf1371fc0b29718c17ebaf6160ec403593c8cbf82b1abb1ee46c5ea0d820 Oct 30 03:47:57.377477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.377369 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:47:57.379097 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:47:57.379066 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c267219_da81_4b92_8015_f49f66e2dd08.slice/crio-24cc7248f907384587615e773d979c19ae1907e322ed9d7a2f623b23f50ba3b5 WatchSource:0}: Error finding container 24cc7248f907384587615e773d979c19ae1907e322ed9d7a2f623b23f50ba3b5: Status 404 returned error can't find the container with id 24cc7248f907384587615e773d979c19ae1907e322ed9d7a2f623b23f50ba3b5 Oct 30 03:47:57.488932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.488894 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" event={"ID":"d0326dcb-0dcc-49f7-a46f-3a1611e438bd","Type":"ContainerStarted","Data":"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6"} Oct 30 03:47:57.489111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.488959 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" event={"ID":"d0326dcb-0dcc-49f7-a46f-3a1611e438bd","Type":"ContainerStarted","Data":"f31eaf1371fc0b29718c17ebaf6160ec403593c8cbf82b1abb1ee46c5ea0d820"} Oct 30 03:47:57.489111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.488991 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:47:57.490799 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.490771 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" event={"ID":"9c267219-da81-4b92-8015-f49f66e2dd08","Type":"ContainerStarted","Data":"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803"} Oct 30 03:47:57.490933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.490813 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" event={"ID":"9c267219-da81-4b92-8015-f49f66e2dd08","Type":"ContainerStarted","Data":"24cc7248f907384587615e773d979c19ae1907e322ed9d7a2f623b23f50ba3b5"} Oct 30 03:47:57.490933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.490845 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:47:57.527842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.527789 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" podStartSLOduration=1.527775568 podStartE2EDuration="1.527775568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:57.526881023 +0000 UTC m=+2705.512000914" watchObservedRunningTime="2026-10-30 03:47:57.527775568 +0000 UTC m=+2705.512895456" Oct 30 03:47:57.574568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.571540 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" podStartSLOduration=1.571506133 podStartE2EDuration="1.571506133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:57.571060742 +0000 UTC m=+2705.556180632" watchObservedRunningTime="2026-10-30 03:47:57.571506133 +0000 UTC m=+2705.556626023" Oct 30 03:47:57.711789 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.711753 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:47:57.711959 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:57.711808 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:47:58.711944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:58.711902 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:47:58.712341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:58.711959 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:47:59.710949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:59.710909 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:47:59.711111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:47:59.710975 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:48:00.711537 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:00.711484 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:48:00.712016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:00.711565 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:48:01.710965 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:01.710931 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:48:01.711143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:01.710984 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 03:48:03.047762 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:03.047725 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:48:03.048966 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:03.048762 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:48:03.050057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:03.050032 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" Oct 30 03:48:04.931461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:04.931428 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:48:04.936162 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:04.936141 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:04.954491 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:04.954465 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:48:04.963301 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:04.963269 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d53363\"" Oct 30 03:48:04.987450 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:04.987419 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d53363-dockercfg-rkscw\"" Oct 30 03:48:05.012599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.012564 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:48:05.018769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.018738 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.031823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.031790 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmxz\" (UniqueName: \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.032140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.031889 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.045149 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.045116 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:48:05.133775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.133740 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpbt\" (UniqueName: \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.133941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.133855 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmxz\" (UniqueName: \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.133941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.133919 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.134031 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.133961 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.134746 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.134720 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.153760 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.153731 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmxz\" (UniqueName: \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\") pod \"payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.234939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.234904 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpbt\" (UniqueName: \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.235145 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.235126 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.235863 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.235844 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.251732 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.251701 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:05.254284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.254264 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpbt\" (UniqueName: \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\") pod \"payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.350354 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.349288 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:05.554944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.554915 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:48:05.555856 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:48:05.555829 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006179d2_d715_4f91_a12e_7944d0842361.slice/crio-e6f8aa57a2cf2b2655161132582d772d6be009ba0293351a140e976a67bf35b1 WatchSource:0}: Error finding container e6f8aa57a2cf2b2655161132582d772d6be009ba0293351a140e976a67bf35b1: Status 404 returned error can't find the container with id e6f8aa57a2cf2b2655161132582d772d6be009ba0293351a140e976a67bf35b1 Oct 30 03:48:05.660344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:05.660316 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:48:05.661420 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:48:05.661395 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576b81e6_4ed6_4e25_806a_a09405ba4679.slice/crio-b13c721183cde644bf61a505f979ee38f64aa5728b9a2ad05e263c630d84a3fc WatchSource:0}: Error finding container b13c721183cde644bf61a505f979ee38f64aa5728b9a2ad05e263c630d84a3fc: Status 404 returned error can't find the container with id b13c721183cde644bf61a505f979ee38f64aa5728b9a2ad05e263c630d84a3fc Oct 30 03:48:06.551127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.551086 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" event={"ID":"006179d2-d715-4f91-a12e-7944d0842361","Type":"ContainerStarted","Data":"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22"} Oct 30 03:48:06.551673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.551145 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" event={"ID":"006179d2-d715-4f91-a12e-7944d0842361","Type":"ContainerStarted","Data":"e6f8aa57a2cf2b2655161132582d772d6be009ba0293351a140e976a67bf35b1"} Oct 30 03:48:06.551673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.551181 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:06.552943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.552907 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" event={"ID":"576b81e6-4ed6-4e25-806a-a09405ba4679","Type":"ContainerStarted","Data":"9e12215b01349b970feaf43ca0b56380cb7b9c02a23120ac1aa12a2d92376e50"} Oct 30 03:48:06.553065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.552963 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:06.553065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.552980 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" event={"ID":"576b81e6-4ed6-4e25-806a-a09405ba4679","Type":"ContainerStarted","Data":"b13c721183cde644bf61a505f979ee38f64aa5728b9a2ad05e263c630d84a3fc"} Oct 30 03:48:06.592597 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.592548 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" podStartSLOduration=2.592533584 podStartE2EDuration="2.592533584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:06.58847332 +0000 UTC m=+2714.573593207" watchObservedRunningTime="2026-10-30 03:48:06.592533584 +0000 UTC m=+2714.577653468" Oct 30 03:48:06.632991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:06.632934 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" podStartSLOduration=2.632919478 podStartE2EDuration="2.632919478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:06.629922238 +0000 UTC m=+2714.615042126" watchObservedRunningTime="2026-10-30 03:48:06.632919478 +0000 UTC m=+2714.618039366" Oct 30 03:48:08.499711 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:08.499682 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:48:08.500143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:08.499747 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:48:11.749638 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:48:11.749594 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:48:17.563462 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:17.563423 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:48:17.564116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:17.563883 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:48:19.108693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:19.108646 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:48:19.109338 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:19.109298 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:48:31.924673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.924616 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:48:31.934113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.934083 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:31.954166 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.954124 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:48:31.960696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.960668 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ae0e75-serving-cert\"" Oct 30 03:48:31.999137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.999107 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:31.999328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.999154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:31.999401 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.999326 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pn9p\" (UniqueName: \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:31.999580 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:31.999550 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.101413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.101276 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pn9p\" (UniqueName: \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.101673 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.101593 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.101752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.101726 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.101813 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.101773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.102555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.102508 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.104280 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.104257 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.104366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.104310 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.123591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.123562 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pn9p\" (UniqueName: \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\") pod \"maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.249364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.249331 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:32.579383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.579351 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:48:32.580040 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:48:32.580012 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38db9cc_d6da_40bb_925b_32a44080ad50.slice/crio-58e6dd4eecd821a161c2d8ade04a085a1e68f33d67646b19eee92cd039d5e8d5 WatchSource:0}: Error finding container 58e6dd4eecd821a161c2d8ade04a085a1e68f33d67646b19eee92cd039d5e8d5: Status 404 returned error can't find the container with id 58e6dd4eecd821a161c2d8ade04a085a1e68f33d67646b19eee92cd039d5e8d5 Oct 30 03:48:32.582067 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.582045 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:48:32.727767 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:32.727731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" event={"ID":"c38db9cc-d6da-40bb-925b-32a44080ad50","Type":"ContainerStarted","Data":"58e6dd4eecd821a161c2d8ade04a085a1e68f33d67646b19eee92cd039d5e8d5"} Oct 30 03:48:33.736212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:33.736177 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" event={"ID":"c38db9cc-d6da-40bb-925b-32a44080ad50","Type":"ContainerStarted","Data":"8eb808cb13702b290eef47823ffcb17091744cfe0806c63fcec583c9fd50f144"} Oct 30 03:48:33.736654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:33.736240 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:33.776589 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:33.776537 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" podStartSLOduration=2.246725154 podStartE2EDuration="2.776510197s" totalImagesPullingTime="529.785043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:33.775442759 +0000 UTC m=+2741.760562648" watchObservedRunningTime="2026-10-30 03:48:33.776510197 +0000 UTC m=+2741.761630085" Oct 30 03:48:39.748122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:39.748091 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:48:40.668453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.668420 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:48:40.674062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.674041 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.700665 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.698698 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d53363-serving-cert\"" Oct 30 03:48:40.711274 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.711233 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.711453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.711324 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntbr\" (UniqueName: \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.711568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.711550 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.711634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.711604 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.712465 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.712438 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:48:40.813074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.813040 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.813512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.813108 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.813512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.813157 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.813512 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.813227 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntbr\" (UniqueName: \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.813902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.813873 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.815756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.815734 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.815871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.815827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.832319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.832295 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntbr\" (UniqueName: \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\") pod \"maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:40.997400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:40.997365 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:41.304038 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:48:41.304004 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f733ab9_e299_4011_a75c_1b6020f2cc38.slice/crio-a5c7aa41933ae8f7e7e948601b9c35772e3f8e8b06be617360149d9c56885b99 WatchSource:0}: Error finding container a5c7aa41933ae8f7e7e948601b9c35772e3f8e8b06be617360149d9c56885b99: Status 404 returned error can't find the container with id a5c7aa41933ae8f7e7e948601b9c35772e3f8e8b06be617360149d9c56885b99 Oct 30 03:48:41.304809 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:41.304780 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:48:41.767006 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:48:41.765648 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:48:41.788950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:41.788915 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" event={"ID":"2f733ab9-e299-4011-a75c-1b6020f2cc38","Type":"ContainerStarted","Data":"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5"} Oct 30 03:48:41.789129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:41.788973 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" event={"ID":"2f733ab9-e299-4011-a75c-1b6020f2cc38","Type":"ContainerStarted","Data":"a5c7aa41933ae8f7e7e948601b9c35772e3f8e8b06be617360149d9c56885b99"} Oct 30 03:48:41.789129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:41.789002 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:48:41.834125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:41.834075 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" podStartSLOduration=1.52281347 podStartE2EDuration="1.83405893s" totalImagesPullingTime="311.24546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:41.830261907 +0000 UTC m=+2749.815381835" watchObservedRunningTime="2026-10-30 03:48:41.83405893 +0000 UTC m=+2749.819178818" Oct 30 03:48:47.801600 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:48:47.801566 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:49:11.750293 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:11.750249 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:49:25.888700 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.888659 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f"] Oct 30 03:49:25.893226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.893206 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:25.907562 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.907493 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:25.907699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.907659 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26wh\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:25.907783 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.907712 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:25.907900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.907878 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:25.923551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.923499 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"routing-overlay\"" Oct 30 03:49:25.923841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.923594 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-441461\"" Oct 30 03:49:25.923841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.923746 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"agc-cred-f55f38b5809a\"" Oct 30 03:49:25.923841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.923796 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"openshift-service-ca.crt\"" Oct 30 03:49:25.924041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.924009 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"payload-processing-external-model-e2e--d3502631-dockercfg-nlc98\"" Oct 30 03:49:25.924681 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.924658 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"kube-root-ca.crt\"" Oct 30 03:49:25.929473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:25.929448 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f"] Oct 30 03:49:26.009280 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.009223 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.009479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.009342 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26wh\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.009479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.009401 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.009640 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.009511 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.009948 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.009924 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-441461/agc-cred-f55f38b5809a: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:26.010024 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.009952 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:26.010087 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.010025 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials podName:c212f435-ffce-45b1-b2e2-5869dc036457 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:26.510004666 +0000 UTC m=+2794.495124534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457") : secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:26.010695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.010669 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.010912 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.010887 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.033548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.033502 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26wh\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.516329 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:26.516294 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:26.516561 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.516458 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-441461/agc-cred-f55f38b5809a: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:26.516561 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.516480 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:26.516701 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:26.516577 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials podName:c212f435-ffce-45b1-b2e2-5869dc036457 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:27.516558037 +0000 UTC m=+2795.501677918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457") : secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:27.528065 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:27.528030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:27.528560 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:27.528163 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-441461/agc-cred-f55f38b5809a: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:27.528560 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:27.528182 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:27.528560 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:27.528242 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials podName:c212f435-ffce-45b1-b2e2-5869dc036457 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:29.528227237 +0000 UTC m=+2797.513347102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457") : secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:28.812754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:28.812712 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f"] Oct 30 03:49:28.813176 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:28.813099 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" podUID="c212f435-ffce-45b1-b2e2-5869dc036457" Oct 30 03:49:29.100132 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.100038 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:29.108943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.108920 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:29.149473 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.149433 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\") pod \"c212f435-ffce-45b1-b2e2-5869dc036457\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " Oct 30 03:49:29.149674 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.149584 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\") pod \"c212f435-ffce-45b1-b2e2-5869dc036457\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " Oct 30 03:49:29.149761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.149746 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\") pod \"c212f435-ffce-45b1-b2e2-5869dc036457\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " Oct 30 03:49:29.149952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.149907 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume" pod "c212f435-ffce-45b1-b2e2-5869dc036457" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:29.150171 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.150148 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume" pod "c212f435-ffce-45b1-b2e2-5869dc036457" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:29.150324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.150310 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:49:29.150378 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.150325 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c212f435-ffce-45b1-b2e2-5869dc036457-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:49:29.151812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.151790 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh" pod "c212f435-ffce-45b1-b2e2-5869dc036457" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457"). InnerVolumeSpecName "kube-api-access-j26wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:29.251951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.251918 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26wh\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-kube-api-access-j26wh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:49:29.556041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:29.556002 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f\" (UID: \"c212f435-ffce-45b1-b2e2-5869dc036457\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:29.556228 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:29.556142 4560 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-441461/agc-cred-f55f38b5809a: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:29.556228 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:29.556162 4560 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f: secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:29.556228 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:29.556217 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials podName:c212f435-ffce-45b1-b2e2-5869dc036457 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:33.556203248 +0000 UTC m=+2801.541323128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" (UID: "c212f435-ffce-45b1-b2e2-5869dc036457") : secret "agc-cred-f55f38b5809a" not found Oct 30 03:49:30.106096 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:30.106060 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f" Oct 30 03:49:30.188903 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:30.188856 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f"] Oct 30 03:49:30.194027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:30.193994 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86qxd4f"] Oct 30 03:49:30.265443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:30.265410 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c212f435-ffce-45b1-b2e2-5869dc036457-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:49:32.156383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:32.156352 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c212f435-ffce-45b1-b2e2-5869dc036457" path="/var/lib/kubelet/pods/c212f435-ffce-45b1-b2e2-5869dc036457/volumes" Oct 30 03:49:36.617902 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.617856 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:49:36.623815 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.623789 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.638232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.638205 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:49:36.679397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.679367 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-35fc87\"" Oct 30 03:49:36.680044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.679906 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-35fc87-dockercfg-lh6t9\"" Oct 30 03:49:36.751117 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.751078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.751318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.751125 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsp94\" (UniqueName: \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.852671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.852637 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.852878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.852684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsp94\" (UniqueName: \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.853445 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.853422 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.872090 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.872031 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsp94\" (UniqueName: \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\") pod \"payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:36.939861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:36.939822 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:37.254888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:37.254855 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:49:37.255044 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:49:37.255019 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e1b1f5_c60c_4869_9f37_ae204e5e86a8.slice/crio-67d8beae25986cc1b2a5684e1009e9c9f1684476a0f3cfd0c91d83037a3751ec WatchSource:0}: Error finding container 67d8beae25986cc1b2a5684e1009e9c9f1684476a0f3cfd0c91d83037a3751ec: Status 404 returned error can't find the container with id 67d8beae25986cc1b2a5684e1009e9c9f1684476a0f3cfd0c91d83037a3751ec Oct 30 03:49:38.159804 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:38.159768 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" event={"ID":"61e1b1f5-c60c-4869-9f37-ae204e5e86a8","Type":"ContainerStarted","Data":"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667"} Oct 30 03:49:38.160196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:38.159825 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" event={"ID":"61e1b1f5-c60c-4869-9f37-ae204e5e86a8","Type":"ContainerStarted","Data":"67d8beae25986cc1b2a5684e1009e9c9f1684476a0f3cfd0c91d83037a3751ec"} Oct 30 03:49:38.160196 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:38.159851 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:49:38.199229 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:38.199176 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" podStartSLOduration=2.19916197 podStartE2EDuration="2.19916197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:38.198096441 +0000 UTC m=+2806.183216334" watchObservedRunningTime="2026-10-30 03:49:38.19916197 +0000 UTC m=+2806.184281857" Oct 30 03:49:41.753179 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:49:41.753132 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:49:49.168968 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:49:49.168931 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:50:03.851603 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.851559 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:50:03.857165 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.857140 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:03.879825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.879797 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e84013-serving-cert\"" Oct 30 03:50:03.885228 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.885194 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:50:03.998246 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.998167 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:03.998418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.998305 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzthj\" (UniqueName: \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:03.998418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.998385 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:03.998418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:03.998407 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.099766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.099724 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzthj\" (UniqueName: \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.099950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.099793 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.099950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.099825 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.100365 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.099991 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.100671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.100637 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.102658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.102601 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.102658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.102604 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.122346 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.122309 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzthj\" (UniqueName: \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\") pod \"maas-api-e2e-inf-b-e84013-67776644f5-967nk\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.175437 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.175407 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:04.488775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:04.488747 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:50:04.490769 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:50:04.490740 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65719913_37b3_45ee_8eef_ef409648552c.slice/crio-bf920917ba0b5a265685704384e72e62e9ddb018fc3aac7a91f23660211e749b WatchSource:0}: Error finding container bf920917ba0b5a265685704384e72e62e9ddb018fc3aac7a91f23660211e749b: Status 404 returned error can't find the container with id bf920917ba0b5a265685704384e72e62e9ddb018fc3aac7a91f23660211e749b Oct 30 03:50:05.343131 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:05.343079 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" event={"ID":"65719913-37b3-45ee-8eef-ef409648552c","Type":"ContainerStarted","Data":"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0"} Oct 30 03:50:05.343648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:05.343146 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" event={"ID":"65719913-37b3-45ee-8eef-ef409648552c","Type":"ContainerStarted","Data":"bf920917ba0b5a265685704384e72e62e9ddb018fc3aac7a91f23660211e749b"} Oct 30 03:50:05.343648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:05.343193 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:05.390634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:05.390579 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" podStartSLOduration=2.038648177 podStartE2EDuration="2.390564468s" totalImagesPullingTime="351.916291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:05.387943541 +0000 UTC m=+2833.373063429" watchObservedRunningTime="2026-10-30 03:50:05.390564468 +0000 UTC m=+2833.375684359" Oct 30 03:50:11.387853 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:11.387808 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:50:11.827570 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:50:11.823976 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:50:13.771834 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.771800 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:50:13.776702 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.776681 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.795619 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.795589 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:50:13.804567 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.804540 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9b8610\"/\"kube-root-ca.crt\"" Oct 30 03:50:13.807845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.807819 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9b8610\"/\"default-dockercfg-mq2mg\"" Oct 30 03:50:13.807984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.807885 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9b8610\"/\"openshift-service-ca.crt\"" Oct 30 03:50:13.807984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.807928 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9b8610\"/\"e2e-usage-sim-de99d2-cert\"" Oct 30 03:50:13.871738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.871707 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.871889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.871872 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfbb\" (UniqueName: \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.973028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.972988 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfbb\" (UniqueName: \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.973206 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.973123 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.975696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.975664 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:13.993513 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:13.993484 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfbb\" (UniqueName: \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\") pod \"e2e-usage-sim-de99d2-669f7b9cff-dj7mt\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:14.090020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:14.089937 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:14.411137 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:14.411105 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:50:14.411271 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:50:14.411246 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18aa12a2_b1d7_4e48_9127_a7f2ef9baccf.slice/crio-fac8aaf3a2694594352840652fed7f3b75e2e8b12c987486a11f16adcd071dec WatchSource:0}: Error finding container fac8aaf3a2694594352840652fed7f3b75e2e8b12c987486a11f16adcd071dec: Status 404 returned error can't find the container with id fac8aaf3a2694594352840652fed7f3b75e2e8b12c987486a11f16adcd071dec Oct 30 03:50:14.441227 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:14.441196 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" event={"ID":"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf","Type":"ContainerStarted","Data":"fac8aaf3a2694594352840652fed7f3b75e2e8b12c987486a11f16adcd071dec"} Oct 30 03:50:15.449442 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:15.449404 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" event={"ID":"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf","Type":"ContainerStarted","Data":"bf4ccf4de108f01624c8096e84649c6aefdb9ef465b7184593a74b182978be38"} Oct 30 03:50:15.449901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:15.449625 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:15.455025 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:15.454999 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:50:15.490885 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:15.490831 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" podStartSLOduration=2.490816418 podStartE2EDuration="2.490816418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:15.488907076 +0000 UTC m=+2843.474026964" watchObservedRunningTime="2026-10-30 03:50:15.490816418 +0000 UTC m=+2843.475936305" Oct 30 03:50:33.984876 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:33.984841 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:50:38.985618 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:38.985548 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" podUID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" containerName="payload-processing" containerID="cri-o://85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7" gracePeriod=25 Oct 30 03:50:39.251477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.251413 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:50:39.300656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.300501 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\") pod \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " Oct 30 03:50:39.300656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.300563 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\") pod \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " Oct 30 03:50:39.300960 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.300931 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume" pod "bc79c25c-0500-4625-93fb-1fdbf8b3c031" (UID: "bc79c25c-0500-4625-93fb-1fdbf8b3c031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:39.301086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.300969 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\") pod \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " Oct 30 03:50:39.301086 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.301060 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\") pod \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\" (UID: \"bc79c25c-0500-4625-93fb-1fdbf8b3c031\") " Oct 30 03:50:39.301272 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.301240 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume" pod "bc79c25c-0500-4625-93fb-1fdbf8b3c031" (UID: "bc79c25c-0500-4625-93fb-1fdbf8b3c031"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:39.301615 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.301595 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:50:39.301696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.301620 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79c25c-0500-4625-93fb-1fdbf8b3c031-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:50:39.303153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.303132 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials" pod "bc79c25c-0500-4625-93fb-1fdbf8b3c031" (UID: "bc79c25c-0500-4625-93fb-1fdbf8b3c031"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:39.303412 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.303393 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm" pod "bc79c25c-0500-4625-93fb-1fdbf8b3c031" (UID: "bc79c25c-0500-4625-93fb-1fdbf8b3c031"). InnerVolumeSpecName "kube-api-access-fk7vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:39.403077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.403040 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7vm\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-kube-api-access-fk7vm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:50:39.403077 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.403073 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79c25c-0500-4625-93fb-1fdbf8b3c031-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:50:39.621033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.620937 4560 generic.go:356] "Generic (PLEG): container finished" podID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" containerID="85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7" exitCode=0 Oct 30 03:50:39.621033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.620985 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" event={"ID":"bc79c25c-0500-4625-93fb-1fdbf8b3c031","Type":"ContainerDied","Data":"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7"} Oct 30 03:50:39.621033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.620992 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" Oct 30 03:50:39.621033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.621025 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9" event={"ID":"bc79c25c-0500-4625-93fb-1fdbf8b3c031","Type":"ContainerDied","Data":"3d14b594eb78ff26ffb334a5d99a4cb7ff5bfcc6a803740d46a160209b504f02"} Oct 30 03:50:39.621358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.621047 4560 scope.go:122] "RemoveContainer" containerID="85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7" Oct 30 03:50:39.632469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.632445 4560 scope.go:122] "RemoveContainer" containerID="85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7" Oct 30 03:50:39.632753 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:50:39.632734 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7\": container with ID starting with 85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7 not found: ID does not exist" containerID="85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7" Oct 30 03:50:39.632802 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.632762 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7"} err="failed to get container status \"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7\": rpc error: code = NotFound desc = could not find container \"85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7\": container with ID starting with 85a57dc0a4c9c3283c8d6728d195ede19033f5d327b69d211a82c5ed629bebe7 not found: ID does not exist" Oct 30 03:50:39.681523 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.681473 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:50:39.693991 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:39.693962 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ms4k9"] Oct 30 03:50:40.154260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:40.154225 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" path="/var/lib/kubelet/pods/bc79c25c-0500-4625-93fb-1fdbf8b3c031/volumes" Oct 30 03:50:41.751960 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:50:41.751924 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:50:59.719867 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.719833 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:50:59.720629 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.720611 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" containerName="payload-processing" Oct 30 03:50:59.720678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.720630 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" containerName="payload-processing" Oct 30 03:50:59.720810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.720799 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc79c25c-0500-4625-93fb-1fdbf8b3c031" containerName="payload-processing" Oct 30 03:50:59.729795 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.729761 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.756614 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.756584 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae0e75\"/\"default-dockercfg-hzb9w\"" Oct 30 03:50:59.758309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.757002 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae0e75\"/\"test-model-ae0e75-kserve-self-signed-certs\"" Oct 30 03:50:59.758309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.757856 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:50:59.764304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764277 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae0e75\"/\"openshift-service-ca.crt\"" Oct 30 03:50:59.764420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764330 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae0e75\"/\"kube-root-ca.crt\"" Oct 30 03:50:59.764420 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764390 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.764510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764478 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tmg\" (UniqueName: \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.764510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764503 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.764624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764592 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.764719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764688 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.764836 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.764744 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866358 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tmg\" (UniqueName: \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866412 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866453 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866511 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866592 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.866888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.866748 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.867384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.867039 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.867384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.867116 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.867384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.867339 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.869181 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.869157 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.869819 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.869800 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:50:59.921683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:50:59.921640 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tmg\" (UniqueName: \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\") pod \"test-model-ae0e75-kserve-794fbd6797-q4hz4\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:00.063751 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:00.063651 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:00.408471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:00.408436 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:51:00.409349 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:00.409322 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66c868d4_47bf_4020_9ab5_666f383724a2.slice/crio-a0bd0e34f131be00bc4522e213e279295ff9081f8e862bc21f655b4ce38ece5a WatchSource:0}: Error finding container a0bd0e34f131be00bc4522e213e279295ff9081f8e862bc21f655b4ce38ece5a: Status 404 returned error can't find the container with id a0bd0e34f131be00bc4522e213e279295ff9081f8e862bc21f655b4ce38ece5a Oct 30 03:51:00.779062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:00.779020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" event={"ID":"66c868d4-47bf-4020-9ab5-666f383724a2","Type":"ContainerStarted","Data":"1515ee78188a1a061bd074b938b2b31a30e05cf27bab5cbfa7528caa2e3097d2"} Oct 30 03:51:00.779467 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:00.779070 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" event={"ID":"66c868d4-47bf-4020-9ab5-666f383724a2","Type":"ContainerStarted","Data":"a0bd0e34f131be00bc4522e213e279295ff9081f8e862bc21f655b4ce38ece5a"} Oct 30 03:51:00.831865 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:00.831809 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" podStartSLOduration=1.831793263 podStartE2EDuration="1.831793263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:00.830393677 +0000 UTC m=+2888.815513566" watchObservedRunningTime="2026-10-30 03:51:00.831793263 +0000 UTC m=+2888.816913150" Oct 30 03:51:01.064742 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.064656 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:01.114251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.114212 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:51:01.119891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.119861 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.143241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.143206 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:51:01.190822 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.190791 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.190828 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.190945 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.190977 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.191002 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.191093 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rtq\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191191 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.191119 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.191216 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.191279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.191253 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.214349 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.214323 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-35fc87-openshift-default-dockercfg-n4t6x\"" Oct 30 03:51:01.292494 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292458 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292511 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292641 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292661 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292712 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292709 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292738 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292809 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rtq\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292838 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.292941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292921 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.293260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.292970 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.293594 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.293565 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.293752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.293690 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.293752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.293726 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.295571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.295444 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.295571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.295499 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.313472 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.313440 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.313639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.313619 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rtq\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.438221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.438139 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:01.770972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.770924 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:51:01.771459 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:01.771427 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1269bc_dd41_4506_850b_9c3afa532ca0.slice/crio-007c1f3a559384eb058446f81cd962d9d1b9266dcabb4efb504427e219a0ae6d WatchSource:0}: Error finding container 007c1f3a559384eb058446f81cd962d9d1b9266dcabb4efb504427e219a0ae6d: Status 404 returned error can't find the container with id 007c1f3a559384eb058446f81cd962d9d1b9266dcabb4efb504427e219a0ae6d Oct 30 03:51:01.775044 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.775006 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:01.775167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.775103 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:01.775167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.775141 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:01.791194 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.791152 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" event={"ID":"9a1269bc-dd41-4506-850b-9c3afa532ca0","Type":"ContainerStarted","Data":"007c1f3a559384eb058446f81cd962d9d1b9266dcabb4efb504427e219a0ae6d"} Oct 30 03:51:01.890848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.890817 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:51:01.907774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.907072 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:01.928176 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:01.928139 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:51:02.001494 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.001457 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e84013-openshift-default-dockercfg-4sn84\"" Oct 30 03:51:02.006142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006114 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006255 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006159 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006339 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006313 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6cj\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006387 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006503 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006611 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006594 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006699 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006685 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006737 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006718 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.006778 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.006747 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108305 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6cj\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108339 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108377 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108417 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108491 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108546 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108588 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.108844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108677 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.109026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.108991 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.109301 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.109272 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.109422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.109313 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.109422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.109381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.109422 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.109407 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.110998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.110976 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.111177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.111162 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.128855 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.128826 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.129180 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.129156 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6cj\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.245098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.245063 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:02.580636 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.580605 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:51:02.581354 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:02.581325 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b97ed1_2c90_4f15_9c87_a7b5901d22dd.slice/crio-f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77 WatchSource:0}: Error finding container f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77: Status 404 returned error can't find the container with id f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77 Oct 30 03:51:02.584309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.584274 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:02.584396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.584340 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:02.584396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.584369 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:02.800670 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.800634 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" event={"ID":"9a1269bc-dd41-4506-850b-9c3afa532ca0","Type":"ContainerStarted","Data":"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d"} Oct 30 03:51:02.803549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.803495 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" event={"ID":"53b97ed1-2c90-4f15-9c87-a7b5901d22dd","Type":"ContainerStarted","Data":"ca2ccdf64b522ce8e57bccb415bdaf0823fc89197f2472166b031fbdd965c50d"} Oct 30 03:51:02.803689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.803561 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" event={"ID":"53b97ed1-2c90-4f15-9c87-a7b5901d22dd","Type":"ContainerStarted","Data":"f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77"} Oct 30 03:51:02.840366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.840288 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:51:02.844947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.844922 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.867539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.865881 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-372480-openshift-default-dockercfg-564ss\"" Oct 30 03:51:02.889200 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.889132 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podStartSLOduration=1.889108315 podStartE2EDuration="1.889108315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.848206122 +0000 UTC m=+2890.833326010" watchObservedRunningTime="2026-10-30 03:51:02.889108315 +0000 UTC m=+2890.874228203" Oct 30 03:51:02.903531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.900423 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:51:02.926906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926765 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.926906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926820 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.926906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926839 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.926906 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926887 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cgl\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.927239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926945 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.927239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.926963 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.927239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.927028 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.927239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.927050 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.927239 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.927076 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:02.943330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:02.941570 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podStartSLOduration=1.94155194 podStartE2EDuration="1.94155194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.937598568 +0000 UTC m=+2890.922718456" watchObservedRunningTime="2026-10-30 03:51:02.94155194 +0000 UTC m=+2890.926671829" Oct 30 03:51:03.027954 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.027920 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.027965 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.027998 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028124 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028194 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028245 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cgl\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028330 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028360 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028534 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028804 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.028929 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.028849 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.029069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.029048 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.029106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.029071 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.031027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.031002 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.031129 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.031104 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.076540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.060898 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cgl\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.076540 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.075071 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.212847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.212419 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:03.256542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.245548 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:03.256542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.247117 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:03.256542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.247175 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:03.438372 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.438337 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:03.439605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.439578 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 03:51:03.439685 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.439638 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 03:51:03.607998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.607967 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:51:03.608266 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:03.608243 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88176770_8a5c_4f52_9928_2d1b0de1386f.slice/crio-2692d3b2a0328182c0ce5f289665ba35ed993e0109f036dd45188ebb0d2c311d WatchSource:0}: Error finding container 2692d3b2a0328182c0ce5f289665ba35ed993e0109f036dd45188ebb0d2c311d: Status 404 returned error can't find the container with id 2692d3b2a0328182c0ce5f289665ba35ed993e0109f036dd45188ebb0d2c311d Oct 30 03:51:03.612062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.611559 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:03.612062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.611641 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:03.612062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.611676 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:51:03.817532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.817240 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" event={"ID":"88176770-8a5c-4f52-9928-2d1b0de1386f","Type":"ContainerStarted","Data":"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d"} Oct 30 03:51:03.817532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.817288 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" event={"ID":"88176770-8a5c-4f52-9928-2d1b0de1386f","Type":"ContainerStarted","Data":"2692d3b2a0328182c0ce5f289665ba35ed993e0109f036dd45188ebb0d2c311d"} Oct 30 03:51:03.899974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:03.893180 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podStartSLOduration=1.8931568890000001 podStartE2EDuration="1.893156889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:03.873488 +0000 UTC m=+2891.858607893" watchObservedRunningTime="2026-10-30 03:51:03.893156889 +0000 UTC m=+2891.878276778" Oct 30 03:51:04.215558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.214614 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:04.215558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.214668 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:04.215558 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.214777 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:04.245621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.245572 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:04.245806 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.245635 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:04.439543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.439472 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 03:51:04.439746 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:04.439567 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 03:51:05.213539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.213417 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:05.213539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.213502 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:05.245715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.245664 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:05.245897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.245733 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:05.438895 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.438847 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 03:51:05.439072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:05.438916 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 03:51:06.215551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.214985 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:06.215551 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.215045 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:06.245734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.245703 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:06.245898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.245749 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:06.441245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.439285 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 03:51:06.441245 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:06.439343 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 03:51:07.214544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.213924 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:07.214544 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.213987 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:07.246105 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.246062 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:07.246484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.246124 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:07.438599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.438561 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 03:51:07.438901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:07.438871 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 03:51:08.225541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.225314 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:08.225541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.225390 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:08.248680 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.248646 4560 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 03:51:08.257957 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.257914 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 03:51:08.444021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.443995 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:08.444571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.444541 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:08.445173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:08.445155 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:51:09.213231 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:09.213199 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:09.213418 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:09.213248 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:09.261106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:09.261071 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:09.275546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:09.263113 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:09.276705 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:09.276677 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:51:10.071542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:10.069306 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:10.088152 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:10.088115 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:10.213896 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:10.213863 4560 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 03:51:10.214110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:10.213920 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 03:51:10.936892 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:10.936860 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:51:11.218972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:11.218942 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:11.219377 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:11.219358 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:11.220102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:11.220086 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:51:11.750651 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:51:11.750616 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:51:19.706642 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.706605 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:51:19.712894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.712873 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.732193 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.732162 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:51:19.732585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.732555 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3cfed7-serving-cert\"" Oct 30 03:51:19.808781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.808742 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.808949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.808891 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.809056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.809033 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.809113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.809082 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52snw\" (UniqueName: \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.910121 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.910080 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.910293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.910160 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52snw\" (UniqueName: \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.910293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.910285 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.910434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.910407 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.910495 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:51:19.910446 4560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-serving-cert: secret "maas-api-e2e-inf-a-3cfed7-serving-cert" not found Oct 30 03:51:19.910591 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:51:19.910532 4560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls podName:6089fe6e-613b-4c04-9926-175201005702 nodeName:}" failed. No retries permitted until 2026-10-30 03:51:20.410496478 +0000 UTC m=+2908.395616356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls") pod "maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" (UID: "6089fe6e-613b-4c04-9926-175201005702") : secret "maas-api-e2e-inf-a-3cfed7-serving-cert" not found Oct 30 03:51:19.910977 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.910946 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.913147 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.913128 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:19.930510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:19.930480 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52snw\" (UniqueName: \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:20.418992 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:20.418947 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:20.424322 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:20.424284 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:20.632131 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:20.632091 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:20.949958 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:20.949927 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:51:20.950285 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:20.949982 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6089fe6e_613b_4c04_9926_175201005702.slice/crio-e5d37984147bc5a9b36dcef25a9cf8a015fbb99be83e8d5c65f436e307b29866 WatchSource:0}: Error finding container e5d37984147bc5a9b36dcef25a9cf8a015fbb99be83e8d5c65f436e307b29866: Status 404 returned error can't find the container with id e5d37984147bc5a9b36dcef25a9cf8a015fbb99be83e8d5c65f436e307b29866 Oct 30 03:51:21.004810 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:21.004773 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" event={"ID":"6089fe6e-613b-4c04-9926-175201005702","Type":"ContainerStarted","Data":"e5d37984147bc5a9b36dcef25a9cf8a015fbb99be83e8d5c65f436e307b29866"} Oct 30 03:51:22.013532 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:22.013489 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" event={"ID":"6089fe6e-613b-4c04-9926-175201005702","Type":"ContainerStarted","Data":"b15422e171d063af892e0c6c02b29de348080af54e42b41e392b577b7176b731"} Oct 30 03:51:22.013901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:22.013623 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:22.057949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:22.057902 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" podStartSLOduration=2.61909891 podStartE2EDuration="3.057889806s" totalImagesPullingTime="438.790896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:22.056648468 +0000 UTC m=+2910.041768356" watchObservedRunningTime="2026-10-30 03:51:22.057889806 +0000 UTC m=+2910.043009693" Oct 30 03:51:26.810251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.810219 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:51:26.814837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.814819 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.833868 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.833829 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:51:26.841364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.841336 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d53363\"/\"openshift-service-ca.crt\"" Oct 30 03:51:26.841572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.841491 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d53363\"/\"default-dockercfg-n8kqn\"" Oct 30 03:51:26.842400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.842373 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d53363\"/\"kube-root-ca.crt\"" Oct 30 03:51:26.842506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.842475 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d53363\"/\"test-model-d53363-kserve-self-signed-certs\"" Oct 30 03:51:26.912284 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912233 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.912570 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912408 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.912624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912583 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.912624 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912613 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77npr\" (UniqueName: \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.912713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912691 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:26.912782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:26.912767 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.013918 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.013881 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.013983 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014064 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014163 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014226 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014189 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77npr\" (UniqueName: \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014234 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014644 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014741 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.014904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.014858 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.016549 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.016495 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.016984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.016962 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.036056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.036025 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77npr\" (UniqueName: \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\") pod \"test-model-d53363-kserve-78544899f-k4t6m\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.130306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.130218 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:27.458081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:27.458048 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:51:27.458958 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:51:27.458928 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a26b4c2_d0da_4cf5_a838_a08d72f48ff2.slice/crio-a70e7edafece4f5c20e1969754c43e8f7bb19a014ed3a56b5d357e6aaa09f1e2 WatchSource:0}: Error finding container a70e7edafece4f5c20e1969754c43e8f7bb19a014ed3a56b5d357e6aaa09f1e2: Status 404 returned error can't find the container with id a70e7edafece4f5c20e1969754c43e8f7bb19a014ed3a56b5d357e6aaa09f1e2 Oct 30 03:51:28.026838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:28.026809 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:51:28.061772 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:28.061737 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" event={"ID":"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2","Type":"ContainerStarted","Data":"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607"} Oct 30 03:51:28.061952 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:28.061779 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" event={"ID":"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2","Type":"ContainerStarted","Data":"a70e7edafece4f5c20e1969754c43e8f7bb19a014ed3a56b5d357e6aaa09f1e2"} Oct 30 03:51:28.114158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:28.114093 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" podStartSLOduration=2.114076669 podStartE2EDuration="2.114076669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:28.110875149 +0000 UTC m=+2916.095995039" watchObservedRunningTime="2026-10-30 03:51:28.114076669 +0000 UTC m=+2916.099196557" Oct 30 03:51:28.130781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:28.130748 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:37.131221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:37.131136 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:37.143752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:37.143720 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:38.141709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:51:38.141679 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:51:41.751211 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:51:41.751169 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:52:11.750990 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:52:11.750950 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:52:16.045630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.045597 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 03:52:16.051509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.051483 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.067889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.067860 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 03:52:16.100106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.100056 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9e0d09\"" Oct 30 03:52:16.105750 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.105700 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9e0d09-dockercfg-cgdr7\"" Oct 30 03:52:16.146862 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.146729 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxwf\" (UniqueName: \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.147695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.147556 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.249408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.249370 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxwf\" (UniqueName: \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.249645 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.249586 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.250326 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.250297 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.268994 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.268963 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxwf\" (UniqueName: \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\") pod \"payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.372848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.372751 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:16.679419 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:16.679379 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 03:52:16.688839 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:52:16.688808 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636fe6a3_c210_4ad2_8f64_80a61125d145.slice/crio-655fd84c801ddec6aa131933dc1fcf37803bcce280fa6379b4751b76024c91c6 WatchSource:0}: Error finding container 655fd84c801ddec6aa131933dc1fcf37803bcce280fa6379b4751b76024c91c6: Status 404 returned error can't find the container with id 655fd84c801ddec6aa131933dc1fcf37803bcce280fa6379b4751b76024c91c6 Oct 30 03:52:17.459894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:17.459846 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" event={"ID":"636fe6a3-c210-4ad2-8f64-80a61125d145","Type":"ContainerStarted","Data":"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682"} Oct 30 03:52:17.460307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:17.459914 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:17.460307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:17.459938 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" event={"ID":"636fe6a3-c210-4ad2-8f64-80a61125d145","Type":"ContainerStarted","Data":"655fd84c801ddec6aa131933dc1fcf37803bcce280fa6379b4751b76024c91c6"} Oct 30 03:52:17.503270 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:17.503216 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" podStartSLOduration=1.5032022 podStartE2EDuration="1.5032022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:17.498716538 +0000 UTC m=+2965.483836425" watchObservedRunningTime="2026-10-30 03:52:17.5032022 +0000 UTC m=+2965.488322124" Oct 30 03:52:28.470554 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:28.470507 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 03:52:41.750760 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:52:41.750724 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:52:49.493654 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.493615 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:52:49.494275 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.493851 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" podUID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" containerName="sim" containerID="cri-o://bf4ccf4de108f01624c8096e84649c6aefdb9ef465b7184593a74b182978be38" gracePeriod=30 Oct 30 03:52:49.761550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.759153 4560 generic.go:356] "Generic (PLEG): container finished" podID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" containerID="bf4ccf4de108f01624c8096e84649c6aefdb9ef465b7184593a74b182978be38" exitCode=0 Oct 30 03:52:49.761550 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.759246 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" event={"ID":"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf","Type":"ContainerDied","Data":"bf4ccf4de108f01624c8096e84649c6aefdb9ef465b7184593a74b182978be38"} Oct 30 03:52:49.866658 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.866624 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:52:49.935166 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.935126 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\" (UniqueName: \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\") pod \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " Oct 30 03:52:49.935390 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.935208 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\" (UniqueName: \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\") pod \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\" (UID: \"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf\") " Oct 30 03:52:49.937869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.937839 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls" pod "18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" (UID: "18aa12a2-b1d7-4e48-9127-a7f2ef9baccf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:52:49.938011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:49.937942 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb" pod "18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" (UID: "18aa12a2-b1d7-4e48-9127-a7f2ef9baccf"). InnerVolumeSpecName "kube-api-access-gdfbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:52:50.037426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.037333 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdfbb\" (UniqueName: \"kubernetes.io/projected/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-kube-api-access-gdfbb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:52:50.037426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.037377 4560 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:52:50.319964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.319876 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:52:50.320694 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.320676 4560 cpu_manager.go:414] "containerMap: removing container" podUID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" containerName="sim" Oct 30 03:52:50.320754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.320695 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" containerName="sim" Oct 30 03:52:50.320840 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.320830 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" containerName="sim" Oct 30 03:52:50.324108 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.324083 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.342706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.342677 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:52:50.350787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.350757 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-f29b9af32f32\"" Oct 30 03:52:50.350932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.350793 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\"" Oct 30 03:52:50.350932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.350807 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\"" Oct 30 03:52:50.351743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.351685 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\"" Oct 30 03:52:50.351743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.351689 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-df757712a33a\"" Oct 30 03:52:50.351743 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.351741 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-mpmb2\"" Oct 30 03:52:50.351993 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.351755 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\"" Oct 30 03:52:50.443395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.443358 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6zd\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.443599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.443582 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.443666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.443625 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.443725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.443705 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.544605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.544570 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6zd\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.544998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.544751 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.544998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.544789 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.544998 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.544819 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.545575 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.545509 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.545756 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.545731 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.547526 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.547499 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.563981 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.563953 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6zd\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.640605 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.640509 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:50.768170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.768130 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" event={"ID":"18aa12a2-b1d7-4e48-9127-a7f2ef9baccf","Type":"ContainerDied","Data":"fac8aaf3a2694594352840652fed7f3b75e2e8b12c987486a11f16adcd071dec"} Oct 30 03:52:50.768364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.768187 4560 scope.go:122] "RemoveContainer" containerID="bf4ccf4de108f01624c8096e84649c6aefdb9ef465b7184593a74b182978be38" Oct 30 03:52:50.768364 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.768352 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt" Oct 30 03:52:50.822824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.822788 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:52:50.826247 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.826219 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-9b8610/e2e-usage-sim-de99d2-669f7b9cff-dj7mt"] Oct 30 03:52:50.952922 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:52:50.952892 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6d270b_f700_4c43_a873_6436a3e7a4bb.slice/crio-54285dff9785fc874b8fd40b0182c5768871a8fa81ccdf24340fb02d2c0d1169 WatchSource:0}: Error finding container 54285dff9785fc874b8fd40b0182c5768871a8fa81ccdf24340fb02d2c0d1169: Status 404 returned error can't find the container with id 54285dff9785fc874b8fd40b0182c5768871a8fa81ccdf24340fb02d2c0d1169 Oct 30 03:52:50.953573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:50.953543 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:52:51.778266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:51.778214 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" event={"ID":"5e6d270b-f700-4c43-a873-6436a3e7a4bb","Type":"ContainerStarted","Data":"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a"} Oct 30 03:52:51.778701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:51.778279 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" event={"ID":"5e6d270b-f700-4c43-a873-6436a3e7a4bb","Type":"ContainerStarted","Data":"54285dff9785fc874b8fd40b0182c5768871a8fa81ccdf24340fb02d2c0d1169"} Oct 30 03:52:51.778701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:51.778310 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:52:51.820373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:51.820323 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" podStartSLOduration=1.820309476 podStartE2EDuration="1.820309476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:51.81672222 +0000 UTC m=+2999.801842108" watchObservedRunningTime="2026-10-30 03:52:51.820309476 +0000 UTC m=+2999.805429417" Oct 30 03:52:52.156220 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:52.156146 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18aa12a2-b1d7-4e48-9127-a7f2ef9baccf" path="/var/lib/kubelet/pods/18aa12a2-b1d7-4e48-9127-a7f2ef9baccf/volumes" Oct 30 03:52:59.813122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.813069 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 03:52:59.820218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.820195 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:52:59.840187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.840159 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9e0d09-serving-cert\"" Oct 30 03:52:59.842334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.842300 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 03:52:59.969034 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.968995 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:52:59.969268 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.969245 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:52:59.969338 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.969304 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9d9l\" (UniqueName: \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:52:59.969441 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:52:59.969419 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.071983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.070971 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.071983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.071227 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.071983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.071299 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9d9l\" (UniqueName: \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.071983 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.071348 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.074690 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.074662 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.075355 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.075333 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.078081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.078058 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.095838 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.095815 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9d9l\" (UniqueName: \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\") pod \"maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.142746 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.142716 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:00.461324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.461291 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 03:53:00.461840 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:53:00.461812 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73ccabf_284e_4670_b12b_0df010cf5aed.slice/crio-e185c09b3ec560b14e95da06b1d68452bbbc349ae891248f32cb4e8865bf490f WatchSource:0}: Error finding container e185c09b3ec560b14e95da06b1d68452bbbc349ae891248f32cb4e8865bf490f: Status 404 returned error can't find the container with id e185c09b3ec560b14e95da06b1d68452bbbc349ae891248f32cb4e8865bf490f Oct 30 03:53:00.846787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:00.846760 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" event={"ID":"d73ccabf-284e-4670-b12b-0df010cf5aed","Type":"ContainerStarted","Data":"e185c09b3ec560b14e95da06b1d68452bbbc349ae891248f32cb4e8865bf490f"} Oct 30 03:53:01.859763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:01.859717 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" event={"ID":"d73ccabf-284e-4670-b12b-0df010cf5aed","Type":"ContainerStarted","Data":"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f"} Oct 30 03:53:01.860548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:01.859936 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:01.904379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:01.904323 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" podStartSLOduration=2.530775751 podStartE2EDuration="2.904307587s" totalImagesPullingTime="373.531836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:01.901680315 +0000 UTC m=+3009.886800204" watchObservedRunningTime="2026-10-30 03:53:01.904307587 +0000 UTC m=+3009.889427516" Oct 30 03:53:02.787907 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:02.787871 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:53:07.872309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:07.872224 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 03:53:11.750956 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:53:11.750914 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:53:15.433072 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:15.433033 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:53:20.434332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.434288 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" podUID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" containerName="payload-processing" containerID="cri-o://314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a" gracePeriod=25 Oct 30 03:53:20.726555 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.726528 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:53:20.832158 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832129 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\") pod \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " Oct 30 03:53:20.832332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832170 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\") pod \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " Oct 30 03:53:20.832332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832322 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\") pod \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " Oct 30 03:53:20.832416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832404 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\") pod \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\" (UID: \"5e6d270b-f700-4c43-a873-6436a3e7a4bb\") " Oct 30 03:53:20.832695 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832661 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume" pod "5e6d270b-f700-4c43-a873-6436a3e7a4bb" (UID: "5e6d270b-f700-4c43-a873-6436a3e7a4bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:20.832864 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.832708 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume" pod "5e6d270b-f700-4c43-a873-6436a3e7a4bb" (UID: "5e6d270b-f700-4c43-a873-6436a3e7a4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:20.833071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.833051 4560 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:53:20.833120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.833075 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6d270b-f700-4c43-a873-6436a3e7a4bb-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:53:20.834351 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.834327 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials" pod "5e6d270b-f700-4c43-a873-6436a3e7a4bb" (UID: "5e6d270b-f700-4c43-a873-6436a3e7a4bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:20.834447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.834392 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd" pod "5e6d270b-f700-4c43-a873-6436a3e7a4bb" (UID: "5e6d270b-f700-4c43-a873-6436a3e7a4bb"). InnerVolumeSpecName "kube-api-access-sd6zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:20.934443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.934400 4560 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:53:20.934443 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:20.934438 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6zd\" (UniqueName: \"kubernetes.io/projected/5e6d270b-f700-4c43-a873-6436a3e7a4bb-kube-api-access-sd6zd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:53:21.003842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.003802 4560 generic.go:356] "Generic (PLEG): container finished" podID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" containerID="314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a" exitCode=0 Oct 30 03:53:21.004164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.003854 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" Oct 30 03:53:21.005545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.003875 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" event={"ID":"5e6d270b-f700-4c43-a873-6436a3e7a4bb","Type":"ContainerDied","Data":"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a"} Oct 30 03:53:21.005545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.005197 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85" event={"ID":"5e6d270b-f700-4c43-a873-6436a3e7a4bb","Type":"ContainerDied","Data":"54285dff9785fc874b8fd40b0182c5768871a8fa81ccdf24340fb02d2c0d1169"} Oct 30 03:53:21.005545 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.005233 4560 scope.go:122] "RemoveContainer" containerID="314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a" Oct 30 03:53:21.040668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.040642 4560 scope.go:122] "RemoveContainer" containerID="314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a" Oct 30 03:53:21.041099 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:53:21.041054 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a\": container with ID starting with 314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a not found: ID does not exist" containerID="314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a" Oct 30 03:53:21.041201 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.041115 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a"} err="failed to get container status \"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a\": rpc error: code = NotFound desc = could not find container \"314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a\": container with ID starting with 314bc60dcc947d6e9a965c5aecb5306a55ad86ab3f8ff9773072e5cf8a50790a not found: ID does not exist" Oct 30 03:53:21.075953 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.075917 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:53:21.086225 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:21.086195 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db9n6v85"] Oct 30 03:53:22.158860 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:22.158826 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" path="/var/lib/kubelet/pods/5e6d270b-f700-4c43-a873-6436a3e7a4bb/volumes" Oct 30 03:53:27.876089 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.876055 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 03:53:27.876879 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.876860 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" containerName="payload-processing" Oct 30 03:53:27.876982 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.876880 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" containerName="payload-processing" Oct 30 03:53:27.877073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.877061 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6d270b-f700-4c43-a873-6436a3e7a4bb" containerName="payload-processing" Oct 30 03:53:27.881397 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.881371 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:27.905110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.905070 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 03:53:27.908188 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.908147 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0393ed\"" Oct 30 03:53:27.908399 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:27.908183 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0393ed-dockercfg-8vhd2\"" Oct 30 03:53:28.042877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.042841 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgn2\" (UniqueName: \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.043081 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.043043 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.144054 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.143965 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.144212 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.144193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgn2\" (UniqueName: \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.144893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.144866 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.164010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.163977 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgn2\" (UniqueName: \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\") pod \"payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.197775 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.197744 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:28.505471 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:28.505444 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 03:53:28.505619 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:53:28.505586 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bce743b_8233_4ac1_91a8_8eb11bc75238.slice/crio-bb6c8ad363d7606eae251c457f21faf63cb04a34063f8fb7649fb4f8a0d91ff5 WatchSource:0}: Error finding container bb6c8ad363d7606eae251c457f21faf63cb04a34063f8fb7649fb4f8a0d91ff5: Status 404 returned error can't find the container with id bb6c8ad363d7606eae251c457f21faf63cb04a34063f8fb7649fb4f8a0d91ff5 Oct 30 03:53:29.069308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:29.069270 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" event={"ID":"4bce743b-8233-4ac1-91a8-8eb11bc75238","Type":"ContainerStarted","Data":"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f"} Oct 30 03:53:29.069781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:29.069330 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" event={"ID":"4bce743b-8233-4ac1-91a8-8eb11bc75238","Type":"ContainerStarted","Data":"bb6c8ad363d7606eae251c457f21faf63cb04a34063f8fb7649fb4f8a0d91ff5"} Oct 30 03:53:29.069781 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:29.069361 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:29.110823 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:29.110771 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" podStartSLOduration=2.110758991 podStartE2EDuration="2.110758991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:29.107236146 +0000 UTC m=+3037.092356034" watchObservedRunningTime="2026-10-30 03:53:29.110758991 +0000 UTC m=+3037.095878880" Oct 30 03:53:40.081366 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.081339 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 03:53:40.329085 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.329043 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 03:53:40.335125 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.335069 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.350733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.350707 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9e0d09-openshift-default-dockercfg-kfh5d\"" Oct 30 03:53:40.356164 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.356133 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 03:53:40.429801 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.429763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.429973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.429831 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.429973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.429877 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.429973 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.430017 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.430151 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4ld\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430205 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.430187 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.430213 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.430321 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.430308 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531431 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531430 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531554 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531648 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4ld\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531720 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531810 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.531924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531912 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.532098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531964 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.532098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.531984 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.532098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.532031 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.532098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.532058 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.532477 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.532452 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.534179 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.534155 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.534416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.534397 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.550667 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.550637 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.550773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.550723 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4ld\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.650114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.650037 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:40.987368 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.987331 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 03:53:40.989555 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:53:40.989512 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0bfe6a_98d6_4e1c_8ca9_f5d347b05bb2.slice/crio-61a5e75d5543ca358b7d64539de9229b94240752b529f2428a282dae7e213f40 WatchSource:0}: Error finding container 61a5e75d5543ca358b7d64539de9229b94240752b529f2428a282dae7e213f40: Status 404 returned error can't find the container with id 61a5e75d5543ca358b7d64539de9229b94240752b529f2428a282dae7e213f40 Oct 30 03:53:40.993168 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.993131 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:53:40.993623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.993208 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:53:40.993623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:40.993237 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:53:41.171710 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:41.171665 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" event={"ID":"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2","Type":"ContainerStarted","Data":"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5"} Oct 30 03:53:41.172197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:41.171735 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" event={"ID":"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2","Type":"ContainerStarted","Data":"61a5e75d5543ca358b7d64539de9229b94240752b529f2428a282dae7e213f40"} Oct 30 03:53:41.213871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:41.213691 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podStartSLOduration=1.213677025 podStartE2EDuration="1.213677025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:41.212988601 +0000 UTC m=+3049.198108489" watchObservedRunningTime="2026-10-30 03:53:41.213677025 +0000 UTC m=+3049.198796914" Oct 30 03:53:41.660541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:41.657689 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:41.776223 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:53:41.776180 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:53:42.675416 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:42.675377 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 03:53:42.675847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:42.675433 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 03:53:43.665543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:43.662921 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 03:53:43.665543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:43.663021 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 03:53:44.673688 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:44.673644 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 03:53:44.674177 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:44.673710 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 03:53:45.668543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:45.665904 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 03:53:45.668543 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:45.666064 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 03:53:46.662541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:46.656921 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 03:53:46.662541 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:46.657000 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 03:53:47.656093 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:47.656062 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:47.656444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:47.656422 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:47.657030 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:47.657013 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 03:53:58.800023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.799981 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:53:58.804904 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.804876 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.833752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.833722 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3cfed7\"/\"openshift-service-ca.crt\"" Oct 30 03:53:58.836692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.836656 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3cfed7\"/\"kube-root-ca.crt\"" Oct 30 03:53:58.836843 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.836744 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3cfed7\"/\"default-dockercfg-rk4v9\"" Oct 30 03:53:58.836946 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.836918 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3cfed7\"/\"test-model-3cfed7-kserve-self-signed-certs\"" Oct 30 03:53:58.838344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.838321 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:53:58.913037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913001 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.913037 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913049 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.913270 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913083 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.913311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913259 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.913408 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913388 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:58.913463 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:58.913450 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4bh\" (UniqueName: \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015396 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015444 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015499 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015594 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015740 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015697 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015773 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4bh\" (UniqueName: \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.015964 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.015918 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.016066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.016032 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.016207 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.016178 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.017919 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.017897 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.018579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.018559 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.035382 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.035353 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4bh\" (UniqueName: \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\") pod \"test-model-3cfed7-kserve-5b6bf6f44c-c4hlj\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.121488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.121396 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:53:59.445693 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:53:59.445661 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:53:59.448199 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:53:59.448172 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf8aa52_21af_4771_b42d_ee5009304f72.slice/crio-46ab64b5b17e8d12a92878181b4bb64cf5b96c7c6cc967a5507c169a3891464f WatchSource:0}: Error finding container 46ab64b5b17e8d12a92878181b4bb64cf5b96c7c6cc967a5507c169a3891464f: Status 404 returned error can't find the container with id 46ab64b5b17e8d12a92878181b4bb64cf5b96c7c6cc967a5507c169a3891464f Oct 30 03:54:00.324531 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:00.324487 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" event={"ID":"ebf8aa52-21af-4771-b42d-ee5009304f72","Type":"ContainerStarted","Data":"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02"} Oct 30 03:54:00.325032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:00.324551 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" event={"ID":"ebf8aa52-21af-4771-b42d-ee5009304f72","Type":"ContainerStarted","Data":"46ab64b5b17e8d12a92878181b4bb64cf5b96c7c6cc967a5507c169a3891464f"} Oct 30 03:54:01.121689 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:01.121656 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:54:09.122336 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:09.122297 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:54:09.135370 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:09.135343 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:54:09.178295 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:09.178231 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" podStartSLOduration=11.178217014 podStartE2EDuration="11.178217014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:00.36971144 +0000 UTC m=+3068.354831329" watchObservedRunningTime="2026-10-30 03:54:09.178217014 +0000 UTC m=+3077.163336901" Oct 30 03:54:09.401187 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:09.401111 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:54:10.425825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.425773 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 03:54:10.431961 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.431935 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.470341 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.470303 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 03:54:10.500181 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500117 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500193 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500369 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500333 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffzp\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500495 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500405 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500492 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500647 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500619 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500709 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500689 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500766 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500740 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.500832 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.500781 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.601916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.601883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602006 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602029 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602083 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffzp\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602124 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602148 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602275 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602311 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602374 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602503 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602485 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602683 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.602725 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.602701 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.603047 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.603024 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.604293 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.604269 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.604585 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.604566 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.624169 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.624135 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.624292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.624202 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffzp\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:10.751379 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:10.751341 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:11.077094 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:54:11.077062 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fddf95_b5e1_4d79_a65f_92fb3b6e1295.slice/crio-91327118b20f8ecad8bd01eb88936e2e3c8071ca9a313a529aaec0acacefc354 WatchSource:0}: Error finding container 91327118b20f8ecad8bd01eb88936e2e3c8071ca9a313a529aaec0acacefc354: Status 404 returned error can't find the container with id 91327118b20f8ecad8bd01eb88936e2e3c8071ca9a313a529aaec0acacefc354 Oct 30 03:54:11.077094 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.077078 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 03:54:11.080041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.080009 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:11.080128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.080075 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:11.080128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.080107 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:11.409434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.409309 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" event={"ID":"64fddf95-b5e1-4d79-a65f-92fb3b6e1295","Type":"ContainerStarted","Data":"628219daa7e2216b097eb5aa841b55065651bc32911611342718c365f40effc0"} Oct 30 03:54:11.409434 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.409422 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" event={"ID":"64fddf95-b5e1-4d79-a65f-92fb3b6e1295","Type":"ContainerStarted","Data":"91327118b20f8ecad8bd01eb88936e2e3c8071ca9a313a529aaec0acacefc354"} Oct 30 03:54:11.454119 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.454049 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podStartSLOduration=1.454028607 podStartE2EDuration="1.454028607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:11.450228744 +0000 UTC m=+3079.435348632" watchObservedRunningTime="2026-10-30 03:54:11.454028607 +0000 UTC m=+3079.439148500" Oct 30 03:54:11.767110 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:11.767071 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:11.777048 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:54:11.777004 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:54:12.770548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:12.768988 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 03:54:12.770548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:12.769074 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 03:54:13.773546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:13.770923 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 03:54:13.773546 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:13.771000 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 03:54:14.675022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.674354 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 03:54:14.681720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.681689 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.682058 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.682024 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 03:54:14.697335 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.697307 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0393ed-serving-cert\"" Oct 30 03:54:14.776757 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.776717 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 03:54:14.777173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.776782 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 03:54:14.785992 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.785961 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.786155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.786012 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.786155 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.786135 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.786307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.786284 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8nw\" (UniqueName: \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.887615 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.887578 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8nw\" (UniqueName: \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.888117 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.888092 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.888259 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.888242 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.888389 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.888372 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.889989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.889959 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.894848 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.894755 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.894994 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.894772 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:14.910016 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:14.909988 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8nw\" (UniqueName: \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\") pod \"maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:15.005041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.005002 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:15.338704 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:54:15.338673 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a2a16_0930_4f67_ae0e_7da4887abbd4.slice/crio-e80d5ea3cc3bb539b56662e5dfae3bccc58163581737e7b476d84bc67022e234 WatchSource:0}: Error finding container e80d5ea3cc3bb539b56662e5dfae3bccc58163581737e7b476d84bc67022e234: Status 404 returned error can't find the container with id e80d5ea3cc3bb539b56662e5dfae3bccc58163581737e7b476d84bc67022e234 Oct 30 03:54:15.340754 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.340726 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:54:15.377573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.347747 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 03:54:15.440259 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.440223 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" event={"ID":"ad3a2a16-0930-4f67-ae0e-7da4887abbd4","Type":"ContainerStarted","Data":"e80d5ea3cc3bb539b56662e5dfae3bccc58163581737e7b476d84bc67022e234"} Oct 30 03:54:15.755951 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.755911 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 03:54:15.756253 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:15.756217 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 03:54:16.473878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:16.473833 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" event={"ID":"ad3a2a16-0930-4f67-ae0e-7da4887abbd4","Type":"ContainerStarted","Data":"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124"} Oct 30 03:54:16.474325 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:16.473967 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:16.516140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:16.516087 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" podStartSLOduration=2.05036911 podStartE2EDuration="2.516073972s" totalImagesPullingTime="465.704862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:16.515018111 +0000 UTC m=+3084.500137998" watchObservedRunningTime="2026-10-30 03:54:16.516073972 +0000 UTC m=+3084.501193859" Oct 30 03:54:16.752986 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:16.752892 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 03:54:16.753260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:16.753226 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 03:54:17.756782 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:17.756749 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:17.757678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:17.757276 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:17.758036 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:17.758014 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 03:54:20.884003 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.883965 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:54:20.922006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.921968 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:54:20.922218 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.922091 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.951106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.951073 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9e0d09\"/\"default-dockercfg-7b5vf\"" Oct 30 03:54:20.951106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.951096 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9e0d09\"/\"openshift-service-ca.crt\"" Oct 30 03:54:20.951319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.951121 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9e0d09\"/\"kube-root-ca.crt\"" Oct 30 03:54:20.951319 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.951179 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9e0d09\"/\"test-model-9e0d09-kserve-self-signed-certs\"" Oct 30 03:54:20.981136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981093 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.981343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981148 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.981343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981246 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.981343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.981635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981614 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:20.981735 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:20.981679 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnc5\" (UniqueName: \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.082995 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083084 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnc5\" (UniqueName: \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083202 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083232 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083268 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083346 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083704 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083767 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.083888 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.083818 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.085657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.085636 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.086183 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.086161 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.102656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.102619 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnc5\" (UniqueName: \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\") pod \"test-model-9e0d09-kserve-bcb9f9db9-6p6cq\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.210539 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.209786 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 03:54:21.223542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.218912 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 03:54:21.223542 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.219034 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.256536 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.240123 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4206d9-serving-cert\"" Oct 30 03:54:21.256536 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.249731 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:21.314232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.298722 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.314232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.298971 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqbj\" (UniqueName: \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.314232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.299086 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.314232 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.299114 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.401631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.401589 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.401856 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.401835 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqbj\" (UniqueName: \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.401973 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.401953 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.402040 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.401986 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.402482 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.402455 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.410622 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.410592 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.411627 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.411600 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.421692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.421665 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqbj\" (UniqueName: \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\") pod \"maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.581633 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.581504 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:21.598837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.598804 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:54:21.600212 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:54:21.600167 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod947bccfd_189e_4b6b_bd92_d5a2d5fda8aa.slice/crio-9541f1a1d528cd8c01b555bd312b8e68db4724901baad945cadbb1d323047a45 WatchSource:0}: Error finding container 9541f1a1d528cd8c01b555bd312b8e68db4724901baad945cadbb1d323047a45: Status 404 returned error can't find the container with id 9541f1a1d528cd8c01b555bd312b8e68db4724901baad945cadbb1d323047a45 Oct 30 03:54:21.909844 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:54:21.909811 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbde9d0_7d76_43d4_abc1_bf2bc2d30e71.slice/crio-27e9d34aa27c9a44832858787ecc4d57c336420df0bc98a84d463ff3debe9bba WatchSource:0}: Error finding container 27e9d34aa27c9a44832858787ecc4d57c336420df0bc98a84d463ff3debe9bba: Status 404 returned error can't find the container with id 27e9d34aa27c9a44832858787ecc4d57c336420df0bc98a84d463ff3debe9bba Oct 30 03:54:21.909844 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:21.909826 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 03:54:22.492204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.492125 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 03:54:22.526566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.526496 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" event={"ID":"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71","Type":"ContainerStarted","Data":"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971"} Oct 30 03:54:22.526753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.526596 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:22.526753 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.526616 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" event={"ID":"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71","Type":"ContainerStarted","Data":"27e9d34aa27c9a44832858787ecc4d57c336420df0bc98a84d463ff3debe9bba"} Oct 30 03:54:22.528308 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.528276 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" event={"ID":"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa","Type":"ContainerStarted","Data":"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06"} Oct 30 03:54:22.528447 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.528322 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" event={"ID":"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa","Type":"ContainerStarted","Data":"9541f1a1d528cd8c01b555bd312b8e68db4724901baad945cadbb1d323047a45"} Oct 30 03:54:22.578869 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.578805 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" podStartSLOduration=1.283610611 podStartE2EDuration="1.578783503s" totalImagesPullingTime="295.172892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:22.575367835 +0000 UTC m=+3090.560487724" watchObservedRunningTime="2026-10-30 03:54:22.578783503 +0000 UTC m=+3090.563903396" Oct 30 03:54:22.620028 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:22.619975 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" podStartSLOduration=2.619959535 podStartE2EDuration="2.619959535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:22.615464532 +0000 UTC m=+3090.600584423" watchObservedRunningTime="2026-10-30 03:54:22.619959535 +0000 UTC m=+3090.605079460" Oct 30 03:54:23.255976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:23.255941 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:25.470736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.470684 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 03:54:25.479022 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.478988 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.533606 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.530722 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 03:54:25.575982 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.575942 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.575982 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.575989 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576030 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576151 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh99r\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576232 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576266 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576281 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.576723 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.576318 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678336 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678302 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh99r\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678437 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678548 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678674 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678557 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678674 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678621 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678716 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678784 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678766 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.678944 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.678858 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.679107 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.679079 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.679318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.679159 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.679318 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.679286 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.679440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.679318 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.679440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.679381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.681170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.681148 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.681414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.681395 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.698276 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.698245 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.698617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.698593 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh99r\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:25.828949 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:25.828859 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:26.152047 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:54:26.152011 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790293ab_1b10_49eb_b35f_89d3ea4fac61.slice/crio-965ccbde7c15bb97bb68e73e120c099158e4bfba5c46f8baa3990d7aec65aae3 WatchSource:0}: Error finding container 965ccbde7c15bb97bb68e73e120c099158e4bfba5c46f8baa3990d7aec65aae3: Status 404 returned error can't find the container with id 965ccbde7c15bb97bb68e73e120c099158e4bfba5c46f8baa3990d7aec65aae3 Oct 30 03:54:26.155394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.155356 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:26.155500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.155443 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:26.155500 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.155494 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 03:54:26.159792 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.159712 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 03:54:26.571453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.571410 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" event={"ID":"790293ab-1b10-49eb-b35f-89d3ea4fac61","Type":"ContainerStarted","Data":"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492"} Oct 30 03:54:26.571995 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.571489 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" event={"ID":"790293ab-1b10-49eb-b35f-89d3ea4fac61","Type":"ContainerStarted","Data":"965ccbde7c15bb97bb68e73e120c099158e4bfba5c46f8baa3990d7aec65aae3"} Oct 30 03:54:26.627864 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.627794 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podStartSLOduration=1.627773818 podStartE2EDuration="1.627773818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:26.624572679 +0000 UTC m=+3094.609692567" watchObservedRunningTime="2026-10-30 03:54:26.627773818 +0000 UTC m=+3094.612893709" Oct 30 03:54:26.829668 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:26.829592 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:27.832330 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:27.832291 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 03:54:27.832774 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:27.832349 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 03:54:28.543653 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:28.543622 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 03:54:28.829557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:28.829452 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 03:54:28.829557 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:28.829511 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 03:54:29.829251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:29.829214 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 03:54:29.829731 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:29.829271 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 03:54:30.829501 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:30.829462 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 03:54:30.829950 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:30.829546 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 03:54:31.271736 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:31.271695 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:31.287240 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:31.287207 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:31.625603 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:31.625509 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:54:31.829499 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:31.829466 4560 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 03:54:31.829671 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:31.829536 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 03:54:32.835111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:32.835078 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:32.835591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:32.835570 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:32.836374 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:54:32.836353 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 03:54:41.750666 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:54:41.750588 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:55:05.794446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:05.794366 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:55:11.751105 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:55:11.751068 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:55:20.822468 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:20.822418 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" containerName="main" containerID="cri-o://1515ee78188a1a061bd074b938b2b31a30e05cf27bab5cbfa7528caa2e3097d2" gracePeriod=45 Oct 30 03:55:20.925470 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:20.925424 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": dial tcp 10.132.0.158:8000: connect: connection refused" Oct 30 03:55:21.057650 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.057614 4560 generic.go:356] "Generic (PLEG): container finished" podID="66c868d4-47bf-4020-9ab5-666f383724a2" containerID="1515ee78188a1a061bd074b938b2b31a30e05cf27bab5cbfa7528caa2e3097d2" exitCode=0 Oct 30 03:55:21.057847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.057691 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" event={"ID":"66c868d4-47bf-4020-9ab5-666f383724a2","Type":"ContainerDied","Data":"1515ee78188a1a061bd074b938b2b31a30e05cf27bab5cbfa7528caa2e3097d2"} Oct 30 03:55:21.111476 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.111440 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:55:21.226798 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.226768 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.226948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.226863 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.226948 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.226924 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.227041 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.226992 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.227076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227025 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:21.227076 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227071 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.227197 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227185 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\" (UniqueName: \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\") pod \"66c868d4-47bf-4020-9ab5-666f383724a2\" (UID: \"66c868d4-47bf-4020-9ab5-666f383724a2\") " Oct 30 03:55:21.227241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227188 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:21.227241 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227207 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:21.227761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227739 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:21.227761 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227761 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:21.227932 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.227773 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:21.229286 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.229262 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:21.229402 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.229373 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "kube-api-access-q4tmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:21.229461 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.229405 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs" pod "66c868d4-47bf-4020-9ab5-666f383724a2" (UID: "66c868d4-47bf-4020-9ab5-666f383724a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:21.329877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.329795 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c868d4-47bf-4020-9ab5-666f383724a2-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:21.329877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.329827 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c868d4-47bf-4020-9ab5-666f383724a2-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:21.329877 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:21.329837 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4tmg\" (UniqueName: \"kubernetes.io/projected/66c868d4-47bf-4020-9ab5-666f383724a2-kube-api-access-q4tmg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:22.067048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.067019 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" Oct 30 03:55:22.067048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.067030 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4" event={"ID":"66c868d4-47bf-4020-9ab5-666f383724a2","Type":"ContainerDied","Data":"a0bd0e34f131be00bc4522e213e279295ff9081f8e862bc21f655b4ce38ece5a"} Oct 30 03:55:22.067571 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.067089 4560 scope.go:122] "RemoveContainer" containerID="1515ee78188a1a061bd074b938b2b31a30e05cf27bab5cbfa7528caa2e3097d2" Oct 30 03:55:22.120826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.120790 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:55:22.133048 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.133010 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae0e75/test-model-ae0e75-kserve-794fbd6797-q4hz4"] Oct 30 03:55:22.155432 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:22.155395 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" path="/var/lib/kubelet/pods/66c868d4-47bf-4020-9ab5-666f383724a2/volumes" Oct 30 03:55:26.080406 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.080364 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 03:55:26.081260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.081239 4560 cpu_manager.go:414] "containerMap: removing container" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" containerName="main" Oct 30 03:55:26.081260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.081260 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" containerName="main" Oct 30 03:55:26.081403 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.081389 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c868d4-47bf-4020-9ab5-666f383724a2" containerName="main" Oct 30 03:55:26.085438 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.085415 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.105564 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.105530 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 03:55:26.111907 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.111873 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3d585\"/\"default-dockercfg-dj45d\"" Oct 30 03:55:26.112074 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.111967 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3d585\"/\"openshift-service-ca.crt\"" Oct 30 03:55:26.119230 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.119189 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3d585\"/\"kube-root-ca.crt\"" Oct 30 03:55:26.119392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.119252 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3d585\"/\"test-model-a3d585-kserve-self-signed-certs\"" Oct 30 03:55:26.210376 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210327 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.210565 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210479 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.210630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210610 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.210704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210688 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.210878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210844 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9pt\" (UniqueName: \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.211021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.210958 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.312122 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312068 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.312334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312252 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.312414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312374 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.312479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312449 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.313124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312638 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9pt\" (UniqueName: \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.313124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.312746 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.313124 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.313071 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.313433 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.313130 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.314811 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.314782 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.329561 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.329487 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.360630 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.346407 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9pt\" (UniqueName: \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.361510 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.361424 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\") pod \"test-model-a3d585-kserve-845b794975-j897k\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.415536 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.415028 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:26.737678 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:26.737645 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 03:55:26.739594 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:55:26.739564 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod055098bc_6674_4e59_8619_afae8f20d297.slice/crio-04bda07c8406243f0ace140aba025f8d5597c4aa6d6f4dab5842ce8d31f9640d WatchSource:0}: Error finding container 04bda07c8406243f0ace140aba025f8d5597c4aa6d6f4dab5842ce8d31f9640d: Status 404 returned error can't find the container with id 04bda07c8406243f0ace140aba025f8d5597c4aa6d6f4dab5842ce8d31f9640d Oct 30 03:55:27.121657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:27.121555 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" event={"ID":"055098bc-6674-4e59-8619-afae8f20d297","Type":"ContainerStarted","Data":"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3"} Oct 30 03:55:27.121657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:27.121607 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" event={"ID":"055098bc-6674-4e59-8619-afae8f20d297","Type":"ContainerStarted","Data":"04bda07c8406243f0ace140aba025f8d5597c4aa6d6f4dab5842ce8d31f9640d"} Oct 30 03:55:27.164897 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:27.164843 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" podStartSLOduration=1.164829269 podStartE2EDuration="1.164829269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:27.160202739 +0000 UTC m=+3155.145322627" watchObservedRunningTime="2026-10-30 03:55:27.164829269 +0000 UTC m=+3155.149949157" Oct 30 03:55:27.416006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:27.415927 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:31.439052 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:31.439012 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:55:31.472351 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:31.472313 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:55:36.415727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:36.415695 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:36.428623 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:36.428594 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:36.439972 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:36.439923 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" containerID="cri-o://fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803" gracePeriod=25 Oct 30 03:55:36.473659 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:36.473626 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" containerID="cri-o://fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6" gracePeriod=25 Oct 30 03:55:37.218111 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:37.218083 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 03:55:38.498842 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:38.498801 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:55:38.499249 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:38.498961 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:55:41.175628 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.175594 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 03:55:41.180701 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.180672 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.205824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.205785 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 03:55:41.208769 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.208471 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4206d9\"/\"test-model-4206d9-kserve-self-signed-certs\"" Oct 30 03:55:41.209613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.208472 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4206d9\"/\"openshift-service-ca.crt\"" Oct 30 03:55:41.209734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.208568 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4206d9\"/\"default-dockercfg-qpf9p\"" Oct 30 03:55:41.209734 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.208624 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4206d9\"/\"kube-root-ca.crt\"" Oct 30 03:55:41.255352 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255293 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.255572 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255491 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.255664 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255641 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.255728 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255710 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.255787 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255738 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.255924 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.255904 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.357827 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.357788 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358017 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.357849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358017 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.357887 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358017 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.357903 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358017 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.357983 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358251 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.358167 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358332 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.358307 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358396 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.358357 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.358828 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.358806 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.360837 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.360812 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.361035 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.361015 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.377883 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.377856 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\") pod \"test-model-4206d9-kserve-866cd87b7c-pxn9b\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.502026 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.501980 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:41.752780 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:55:41.752697 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:55:41.831328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.831291 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 03:55:41.831328 ip-10-0-136-97 kubenswrapper[4560]: W1030 03:55:41.831319 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b2c191_18bc_4974_b952_155fd3761ff9.slice/crio-0da57678695a2cf465a29cac587c89ee5a9bccf76f6cb20ff1096308a0563b13 WatchSource:0}: Error finding container 0da57678695a2cf465a29cac587c89ee5a9bccf76f6cb20ff1096308a0563b13: Status 404 returned error can't find the container with id 0da57678695a2cf465a29cac587c89ee5a9bccf76f6cb20ff1096308a0563b13 Oct 30 03:55:41.881942 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.881763 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:55:41.882039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:41.882000 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" podUID="c38db9cc-d6da-40bb-925b-32a44080ad50" containerName="maas-api" containerID="cri-o://8eb808cb13702b290eef47823ffcb17091744cfe0806c63fcec583c9fd50f144" gracePeriod=30 Oct 30 03:55:42.256084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.255324 4560 generic.go:356] "Generic (PLEG): container finished" podID="c38db9cc-d6da-40bb-925b-32a44080ad50" containerID="8eb808cb13702b290eef47823ffcb17091744cfe0806c63fcec583c9fd50f144" exitCode=0 Oct 30 03:55:42.256084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.255430 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" event={"ID":"c38db9cc-d6da-40bb-925b-32a44080ad50","Type":"ContainerDied","Data":"8eb808cb13702b290eef47823ffcb17091744cfe0806c63fcec583c9fd50f144"} Oct 30 03:55:42.257780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.257745 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" event={"ID":"b9b2c191-18bc-4974-b952-155fd3761ff9","Type":"ContainerStarted","Data":"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793"} Oct 30 03:55:42.257928 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.257791 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" event={"ID":"b9b2c191-18bc-4974-b952-155fd3761ff9","Type":"ContainerStarted","Data":"0da57678695a2cf465a29cac587c89ee5a9bccf76f6cb20ff1096308a0563b13"} Oct 30 03:55:42.339140 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.339081 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" podStartSLOduration=1.33905892 podStartE2EDuration="1.33905892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:42.31314286 +0000 UTC m=+3170.298262752" watchObservedRunningTime="2026-10-30 03:55:42.33905892 +0000 UTC m=+3170.324178809" Oct 30 03:55:42.502292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.502037 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:42.504889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.504864 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:55:42.596573 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.596500 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\" (UniqueName: \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\") pod \"c38db9cc-d6da-40bb-925b-32a44080ad50\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " Oct 30 03:55:42.596812 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.596675 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\") pod \"c38db9cc-d6da-40bb-925b-32a44080ad50\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " Oct 30 03:55:42.596891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.596824 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\") pod \"c38db9cc-d6da-40bb-925b-32a44080ad50\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " Oct 30 03:55:42.596891 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.596860 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\") pod \"c38db9cc-d6da-40bb-925b-32a44080ad50\" (UID: \"c38db9cc-d6da-40bb-925b-32a44080ad50\") " Oct 30 03:55:42.597358 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.597326 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca" pod "c38db9cc-d6da-40bb-925b-32a44080ad50" (UID: "c38db9cc-d6da-40bb-925b-32a44080ad50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:55:42.597706 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.597679 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38db9cc-d6da-40bb-925b-32a44080ad50-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.599429 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.599316 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls" pod "c38db9cc-d6da-40bb-925b-32a44080ad50" (UID: "c38db9cc-d6da-40bb-925b-32a44080ad50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:42.599831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.599801 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p" pod "c38db9cc-d6da-40bb-925b-32a44080ad50" (UID: "c38db9cc-d6da-40bb-925b-32a44080ad50"). InnerVolumeSpecName "kube-api-access-7pn9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:42.599915 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.599887 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs" pod "c38db9cc-d6da-40bb-925b-32a44080ad50" (UID: "c38db9cc-d6da-40bb-925b-32a44080ad50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:42.699161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.699082 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.699161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.699113 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pn9p\" (UniqueName: \"kubernetes.io/projected/c38db9cc-d6da-40bb-925b-32a44080ad50-kube-api-access-7pn9p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.699161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:42.699122 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38db9cc-d6da-40bb-925b-32a44080ad50-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:43.266599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:43.266564 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" Oct 30 03:55:43.267116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:43.266563 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp" event={"ID":"c38db9cc-d6da-40bb-925b-32a44080ad50","Type":"ContainerDied","Data":"58e6dd4eecd821a161c2d8ade04a085a1e68f33d67646b19eee92cd039d5e8d5"} Oct 30 03:55:43.267116 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:43.266724 4560 scope.go:122] "RemoveContainer" containerID="8eb808cb13702b290eef47823ffcb17091744cfe0806c63fcec583c9fd50f144" Oct 30 03:55:43.322691 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:43.322655 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:55:43.335192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:43.335160 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae0e75-66cc8bdbb4-bsfkp"] Oct 30 03:55:44.159610 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:44.159578 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38db9cc-d6da-40bb-925b-32a44080ad50" path="/var/lib/kubelet/pods/c38db9cc-d6da-40bb-925b-32a44080ad50/volumes" Oct 30 03:55:46.459921 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:46.459887 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:55:48.498763 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:48.498723 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:55:48.499263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:48.498724 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:55:51.503010 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:51.502974 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:51.516242 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:51.516214 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:52.406835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:52.406806 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 03:55:56.780234 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.780195 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:55:56.816796 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.816769 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:55:56.924566 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.924491 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\" (UniqueName: \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\") pod \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " Oct 30 03:55:56.924745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.924597 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\" (UniqueName: \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\") pod \"9c267219-da81-4b92-8015-f49f66e2dd08\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " Oct 30 03:55:56.924745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.924714 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\") pod \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\" (UID: \"d0326dcb-0dcc-49f7-a46f-3a1611e438bd\") " Oct 30 03:55:56.924833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.924758 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\") pod \"9c267219-da81-4b92-8015-f49f66e2dd08\" (UID: \"9c267219-da81-4b92-8015-f49f66e2dd08\") " Oct 30 03:55:56.925182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.925156 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume" pod "d0326dcb-0dcc-49f7-a46f-3a1611e438bd" (UID: "d0326dcb-0dcc-49f7-a46f-3a1611e438bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:55:56.925182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.925157 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume" pod "9c267219-da81-4b92-8015-f49f66e2dd08" (UID: "9c267219-da81-4b92-8015-f49f66e2dd08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:55:56.925444 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.925428 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:56.925497 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.925447 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c267219-da81-4b92-8015-f49f66e2dd08-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:56.926713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.926691 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst" pod "d0326dcb-0dcc-49f7-a46f-3a1611e438bd" (UID: "d0326dcb-0dcc-49f7-a46f-3a1611e438bd"). InnerVolumeSpecName "kube-api-access-6chst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:56.926916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:56.926895 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6" pod "9c267219-da81-4b92-8015-f49f66e2dd08" (UID: "9c267219-da81-4b92-8015-f49f66e2dd08"). InnerVolumeSpecName "kube-api-access-rxng6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:57.027269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.027174 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chst\" (UniqueName: \"kubernetes.io/projected/d0326dcb-0dcc-49f7-a46f-3a1611e438bd-kube-api-access-6chst\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:57.027269 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.027215 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxng6\" (UniqueName: \"kubernetes.io/projected/9c267219-da81-4b92-8015-f49f66e2dd08-kube-api-access-rxng6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:55:57.399563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.399446 4560 generic.go:356] "Generic (PLEG): container finished" podID="9c267219-da81-4b92-8015-f49f66e2dd08" containerID="fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803" exitCode=0 Oct 30 03:55:57.399563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.399492 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" Oct 30 03:55:57.399563 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.399534 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" event={"ID":"9c267219-da81-4b92-8015-f49f66e2dd08","Type":"ContainerDied","Data":"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803"} Oct 30 03:55:57.399824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.399592 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt" event={"ID":"9c267219-da81-4b92-8015-f49f66e2dd08","Type":"ContainerDied","Data":"24cc7248f907384587615e773d979c19ae1907e322ed9d7a2f623b23f50ba3b5"} Oct 30 03:55:57.399824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.399621 4560 scope.go:122] "RemoveContainer" containerID="fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803" Oct 30 03:55:57.401173 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.401150 4560 generic.go:356] "Generic (PLEG): container finished" podID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerID="fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6" exitCode=0 Oct 30 03:55:57.401314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.401207 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" Oct 30 03:55:57.401314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.401247 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" event={"ID":"d0326dcb-0dcc-49f7-a46f-3a1611e438bd","Type":"ContainerDied","Data":"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6"} Oct 30 03:55:57.401314 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.401310 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c" event={"ID":"d0326dcb-0dcc-49f7-a46f-3a1611e438bd","Type":"ContainerDied","Data":"f31eaf1371fc0b29718c17ebaf6160ec403593c8cbf82b1abb1ee46c5ea0d820"} Oct 30 03:55:57.411170 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.411152 4560 scope.go:122] "RemoveContainer" containerID="fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803" Oct 30 03:55:57.411527 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:55:57.411483 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803\": container with ID starting with fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803 not found: ID does not exist" containerID="fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803" Oct 30 03:55:57.411581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.411526 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803"} err="failed to get container status \"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803\": rpc error: code = NotFound desc = could not find container \"fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803\": container with ID starting with fa3b0154a4897d2261a209b410abdba2bae2059e24f09f1195007ec2130ea803 not found: ID does not exist" Oct 30 03:55:57.411581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.411549 4560 scope.go:122] "RemoveContainer" containerID="fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6" Oct 30 03:55:57.422250 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.422228 4560 scope.go:122] "RemoveContainer" containerID="fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6" Oct 30 03:55:57.422596 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:55:57.422568 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6\": container with ID starting with fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6 not found: ID does not exist" containerID="fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6" Oct 30 03:55:57.422704 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.422603 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6"} err="failed to get container status \"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6\": rpc error: code = NotFound desc = could not find container \"fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6\": container with ID starting with fd982c1b614b876104353c309df580b9bc6c828397bfd1d88db6ca76f36c33f6 not found: ID does not exist" Oct 30 03:55:57.455292 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.455256 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:55:57.467032 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.467000 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae0e75-799dd5b95f-v7n4c"] Oct 30 03:55:57.505088 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.505051 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:55:57.510656 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:57.510628 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae0e75-c9bff5778-g64mt"] Oct 30 03:55:58.155788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:58.155756 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" path="/var/lib/kubelet/pods/9c267219-da81-4b92-8015-f49f66e2dd08/volumes" Oct 30 03:55:58.156411 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:55:58.156389 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" path="/var/lib/kubelet/pods/d0326dcb-0dcc-49f7-a46f-3a1611e438bd/volumes" Oct 30 03:56:01.491916 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.491871 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" podUID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" containerName="main" containerID="cri-o://fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607" gracePeriod=45 Oct 30 03:56:01.789579 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.789549 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:56:01.907930 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.907896 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\" (UniqueName: \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.908136 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908054 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.908202 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908180 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.908260 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908211 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.908315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908296 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.909106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908396 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\") pod \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\" (UID: \"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2\") " Oct 30 03:56:01.909106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.908744 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:01.909391 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.909367 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:01.909702 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.909681 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:01.910102 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.910078 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:01.912291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.912257 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "kube-api-access-77npr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:01.913268 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.913241 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:01.914221 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:01.913792 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm" pod "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" (UID: "9a26b4c2-d0da-4cf5-a838-a08d72f48ff2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:02.011552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.011450 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:02.011552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.011486 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:02.011552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.011497 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:02.011552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.011506 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:02.011552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.011528 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77npr\" (UniqueName: \"kubernetes.io/projected/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2-kube-api-access-77npr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:02.445831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.445744 4560 generic.go:356] "Generic (PLEG): container finished" podID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" containerID="fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607" exitCode=0 Oct 30 03:56:02.445831 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.445792 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" Oct 30 03:56:02.446069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.445825 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" event={"ID":"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2","Type":"ContainerDied","Data":"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607"} Oct 30 03:56:02.446069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.445866 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m" event={"ID":"9a26b4c2-d0da-4cf5-a838-a08d72f48ff2","Type":"ContainerDied","Data":"a70e7edafece4f5c20e1969754c43e8f7bb19a014ed3a56b5d357e6aaa09f1e2"} Oct 30 03:56:02.446069 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.445883 4560 scope.go:122] "RemoveContainer" containerID="fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607" Oct 30 03:56:02.456686 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.456665 4560 scope.go:122] "RemoveContainer" containerID="fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607" Oct 30 03:56:02.456963 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:02.456939 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607\": container with ID starting with fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607 not found: ID does not exist" containerID="fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607" Oct 30 03:56:02.457045 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.456970 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607"} err="failed to get container status \"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607\": rpc error: code = NotFound desc = could not find container \"fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607\": container with ID starting with fd6fbdd2e3c0f5b5f43accd6390a7fbdb191d605915bad02133a1abbd9edc607 not found: ID does not exist" Oct 30 03:56:02.500817 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.500783 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:56:02.513404 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:02.513368 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d53363/test-model-d53363-kserve-78544899f-k4t6m"] Oct 30 03:56:04.156413 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:04.156329 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" path="/var/lib/kubelet/pods/9a26b4c2-d0da-4cf5-a838-a08d72f48ff2/volumes" Oct 30 03:56:04.246881 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:04.246845 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:56:04.279747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:04.279716 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:56:09.248056 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.248011 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" podUID="576b81e6-4ed6-4e25-806a-a09405ba4679" containerName="payload-processing" containerID="cri-o://9e12215b01349b970feaf43ca0b56380cb7b9c02a23120ac1aa12a2d92376e50" gracePeriod=25 Oct 30 03:56:09.280674 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.280619 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" containerID="cri-o://5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22" gracePeriod=25 Oct 30 03:56:09.502858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.502779 4560 generic.go:356] "Generic (PLEG): container finished" podID="576b81e6-4ed6-4e25-806a-a09405ba4679" containerID="9e12215b01349b970feaf43ca0b56380cb7b9c02a23120ac1aa12a2d92376e50" exitCode=0 Oct 30 03:56:09.502858 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.502836 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" event={"ID":"576b81e6-4ed6-4e25-806a-a09405ba4679","Type":"ContainerDied","Data":"9e12215b01349b970feaf43ca0b56380cb7b9c02a23120ac1aa12a2d92376e50"} Oct 30 03:56:09.543331 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.543306 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:56:09.634062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.634015 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\" (UniqueName: \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\") pod \"576b81e6-4ed6-4e25-806a-a09405ba4679\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " Oct 30 03:56:09.634243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.634216 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\") pod \"576b81e6-4ed6-4e25-806a-a09405ba4679\" (UID: \"576b81e6-4ed6-4e25-806a-a09405ba4679\") " Oct 30 03:56:09.634621 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.634593 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume" pod "576b81e6-4ed6-4e25-806a-a09405ba4679" (UID: "576b81e6-4ed6-4e25-806a-a09405ba4679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:09.635095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.635068 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576b81e6-4ed6-4e25-806a-a09405ba4679-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:09.636309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.636287 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt" pod "576b81e6-4ed6-4e25-806a-a09405ba4679" (UID: "576b81e6-4ed6-4e25-806a-a09405ba4679"). InnerVolumeSpecName "kube-api-access-jxpbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:09.736635 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:09.736595 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxpbt\" (UniqueName: \"kubernetes.io/projected/576b81e6-4ed6-4e25-806a-a09405ba4679-kube-api-access-jxpbt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:10.511143 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:10.511108 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" Oct 30 03:56:10.511661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:10.511145 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r" event={"ID":"576b81e6-4ed6-4e25-806a-a09405ba4679","Type":"ContainerDied","Data":"b13c721183cde644bf61a505f979ee38f64aa5728b9a2ad05e263c630d84a3fc"} Oct 30 03:56:10.511661 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:10.511217 4560 scope.go:122] "RemoveContainer" containerID="9e12215b01349b970feaf43ca0b56380cb7b9c02a23120ac1aa12a2d92376e50" Oct 30 03:56:10.564414 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:10.564385 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:56:10.576446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:10.576416 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d53363-54c7d449b4-g9v6r"] Oct 30 03:56:11.751965 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:11.751931 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:56:12.166334 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:12.166255 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576b81e6-4ed6-4e25-806a-a09405ba4679" path="/var/lib/kubelet/pods/576b81e6-4ed6-4e25-806a-a09405ba4679/volumes" Oct 30 03:56:17.562068 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:17.562031 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:56:27.561303 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:27.561263 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:56:29.575446 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.575416 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:56:29.657008 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.656972 4560 generic.go:356] "Generic (PLEG): container finished" podID="006179d2-d715-4f91-a12e-7944d0842361" containerID="5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22" exitCode=0 Oct 30 03:56:29.657208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.657038 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" Oct 30 03:56:29.657208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.657058 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" event={"ID":"006179d2-d715-4f91-a12e-7944d0842361","Type":"ContainerDied","Data":"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22"} Oct 30 03:56:29.657208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.657125 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf" event={"ID":"006179d2-d715-4f91-a12e-7944d0842361","Type":"ContainerDied","Data":"e6f8aa57a2cf2b2655161132582d772d6be009ba0293351a140e976a67bf35b1"} Oct 30 03:56:29.657208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.657160 4560 scope.go:122] "RemoveContainer" containerID="5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22" Oct 30 03:56:29.670400 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.670381 4560 scope.go:122] "RemoveContainer" containerID="5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22" Oct 30 03:56:29.670748 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:29.670713 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22\": container with ID starting with 5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22 not found: ID does not exist" containerID="5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22" Oct 30 03:56:29.670847 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.670757 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22"} err="failed to get container status \"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22\": rpc error: code = NotFound desc = could not find container \"5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22\": container with ID starting with 5a4c00299799007fe093f246acdba9e6f745aeff931681dda0c41547bc944e22 not found: ID does not exist" Oct 30 03:56:29.690926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.690887 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\" (UniqueName: \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\") pod \"006179d2-d715-4f91-a12e-7944d0842361\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " Oct 30 03:56:29.691061 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.690990 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\") pod \"006179d2-d715-4f91-a12e-7944d0842361\" (UID: \"006179d2-d715-4f91-a12e-7944d0842361\") " Oct 30 03:56:29.691359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.691332 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume" pod "006179d2-d715-4f91-a12e-7944d0842361" (UID: "006179d2-d715-4f91-a12e-7944d0842361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:29.691747 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.691729 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006179d2-d715-4f91-a12e-7944d0842361-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:29.693084 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.693061 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz" pod "006179d2-d715-4f91-a12e-7944d0842361" (UID: "006179d2-d715-4f91-a12e-7944d0842361"). InnerVolumeSpecName "kube-api-access-lqmxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:29.793209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:29.793132 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmxz\" (UniqueName: \"kubernetes.io/projected/006179d2-d715-4f91-a12e-7944d0842361-kube-api-access-lqmxz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:30.005826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:30.005777 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:56:30.009489 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:30.009461 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d53363-54cbdbd459-lmqtf"] Oct 30 03:56:30.156773 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:30.156691 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006179d2-d715-4f91-a12e-7944d0842361" path="/var/lib/kubelet/pods/006179d2-d715-4f91-a12e-7944d0842361/volumes" Oct 30 03:56:39.089900 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.089854 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:56:39.090816 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.090160 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" podUID="65719913-37b3-45ee-8eef-ef409648552c" containerName="maas-api" containerID="cri-o://e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0" gracePeriod=30 Oct 30 03:56:39.470498 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.470459 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:56:39.541315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.541281 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\") pod \"65719913-37b3-45ee-8eef-ef409648552c\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " Oct 30 03:56:39.541509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.541390 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\" (UniqueName: \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\") pod \"65719913-37b3-45ee-8eef-ef409648552c\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " Oct 30 03:56:39.541509 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.541449 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\") pod \"65719913-37b3-45ee-8eef-ef409648552c\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " Oct 30 03:56:39.541666 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.541538 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\") pod \"65719913-37b3-45ee-8eef-ef409648552c\" (UID: \"65719913-37b3-45ee-8eef-ef409648552c\") " Oct 30 03:56:39.541933 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.541892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca" pod "65719913-37b3-45ee-8eef-ef409648552c" (UID: "65719913-37b3-45ee-8eef-ef409648552c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:39.542287 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.542266 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65719913-37b3-45ee-8eef-ef409648552c-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:39.545962 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.545009 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls" pod "65719913-37b3-45ee-8eef-ef409648552c" (UID: "65719913-37b3-45ee-8eef-ef409648552c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:39.575569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.568156 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs" pod "65719913-37b3-45ee-8eef-ef409648552c" (UID: "65719913-37b3-45ee-8eef-ef409648552c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:39.575569 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.568159 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj" pod "65719913-37b3-45ee-8eef-ef409648552c" (UID: "65719913-37b3-45ee-8eef-ef409648552c"). InnerVolumeSpecName "kube-api-access-vzthj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:39.644870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.644836 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:39.644870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.644866 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzthj\" (UniqueName: \"kubernetes.io/projected/65719913-37b3-45ee-8eef-ef409648552c-kube-api-access-vzthj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:39.644870 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.644877 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65719913-37b3-45ee-8eef-ef409648552c-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:39.733306 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.733274 4560 generic.go:356] "Generic (PLEG): container finished" podID="65719913-37b3-45ee-8eef-ef409648552c" containerID="e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0" exitCode=0 Oct 30 03:56:39.733526 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.733327 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" Oct 30 03:56:39.733526 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.733360 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" event={"ID":"65719913-37b3-45ee-8eef-ef409648552c","Type":"ContainerDied","Data":"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0"} Oct 30 03:56:39.733526 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.733416 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk" event={"ID":"65719913-37b3-45ee-8eef-ef409648552c","Type":"ContainerDied","Data":"bf920917ba0b5a265685704384e72e62e9ddb018fc3aac7a91f23660211e749b"} Oct 30 03:56:39.733526 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.733443 4560 scope.go:122] "RemoveContainer" containerID="e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0" Oct 30 03:56:39.745451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.745431 4560 scope.go:122] "RemoveContainer" containerID="e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0" Oct 30 03:56:39.745769 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:39.745749 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0\": container with ID starting with e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0 not found: ID does not exist" containerID="e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0" Oct 30 03:56:39.745845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.745778 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0"} err="failed to get container status \"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0\": rpc error: code = NotFound desc = could not find container \"e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0\": container with ID starting with e8959d0fd35fbadc0e0495338d11485d98ae9c769cc9ac5ad30cd99c176e3dc0 not found: ID does not exist" Oct 30 03:56:39.789261 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.789224 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:56:39.792827 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:39.792801 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e84013-67776644f5-967nk"] Oct 30 03:56:40.155337 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:40.155255 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65719913-37b3-45ee-8eef-ef409648552c" path="/var/lib/kubelet/pods/65719913-37b3-45ee-8eef-ef409648552c/volumes" Oct 30 03:56:41.147825 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.147791 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:56:41.148021 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.148001 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" podUID="2f733ab9-e299-4011-a75c-1b6020f2cc38" containerName="maas-api" containerID="cri-o://dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5" gracePeriod=30 Oct 30 03:56:41.542057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.542026 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:56:41.588113 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.588077 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\" (UniqueName: \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\") pod \"2f733ab9-e299-4011-a75c-1b6020f2cc38\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " Oct 30 03:56:41.591145 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.591102 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\") pod \"2f733ab9-e299-4011-a75c-1b6020f2cc38\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " Oct 30 03:56:41.591439 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.591406 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\") pod \"2f733ab9-e299-4011-a75c-1b6020f2cc38\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " Oct 30 03:56:41.591719 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.591699 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\") pod \"2f733ab9-e299-4011-a75c-1b6020f2cc38\" (UID: \"2f733ab9-e299-4011-a75c-1b6020f2cc38\") " Oct 30 03:56:41.592453 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.592410 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca" pod "2f733ab9-e299-4011-a75c-1b6020f2cc38" (UID: "2f733ab9-e299-4011-a75c-1b6020f2cc38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:41.593362 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.593327 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f733ab9-e299-4011-a75c-1b6020f2cc38-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:41.599281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.596835 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs" pod "2f733ab9-e299-4011-a75c-1b6020f2cc38" (UID: "2f733ab9-e299-4011-a75c-1b6020f2cc38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:41.606587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.602709 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr" pod "2f733ab9-e299-4011-a75c-1b6020f2cc38" (UID: "2f733ab9-e299-4011-a75c-1b6020f2cc38"). InnerVolumeSpecName "kube-api-access-mntbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:41.606587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.605976 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls" pod "2f733ab9-e299-4011-a75c-1b6020f2cc38" (UID: "2f733ab9-e299-4011-a75c-1b6020f2cc38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:41.694788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.694753 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntbr\" (UniqueName: \"kubernetes.io/projected/2f733ab9-e299-4011-a75c-1b6020f2cc38-kube-api-access-mntbr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:41.694788 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.694786 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:41.695027 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.694799 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f733ab9-e299-4011-a75c-1b6020f2cc38-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:41.753683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.753589 4560 generic.go:356] "Generic (PLEG): container finished" podID="2f733ab9-e299-4011-a75c-1b6020f2cc38" containerID="dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5" exitCode=0 Oct 30 03:56:41.753683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.753635 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" Oct 30 03:56:41.753683 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.753668 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" event={"ID":"2f733ab9-e299-4011-a75c-1b6020f2cc38","Type":"ContainerDied","Data":"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5"} Oct 30 03:56:41.753943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.753723 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8" event={"ID":"2f733ab9-e299-4011-a75c-1b6020f2cc38","Type":"ContainerDied","Data":"a5c7aa41933ae8f7e7e948601b9c35772e3f8e8b06be617360149d9c56885b99"} Oct 30 03:56:41.753943 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.753745 4560 scope.go:122] "RemoveContainer" containerID="dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5" Oct 30 03:56:41.756232 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:41.756203 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:56:41.766020 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.766002 4560 scope.go:122] "RemoveContainer" containerID="dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5" Oct 30 03:56:41.766295 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:56:41.766279 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5\": container with ID starting with dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5 not found: ID does not exist" containerID="dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5" Oct 30 03:56:41.766373 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.766304 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5"} err="failed to get container status \"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5\": rpc error: code = NotFound desc = could not find container \"dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5\": container with ID starting with dc21e6a7eae4e5f7b836c35fb3512d5c8a1a73c25d99e7d296f181d73958f2f5 not found: ID does not exist" Oct 30 03:56:41.807505 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.807457 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:56:41.811395 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:41.811371 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d53363-6b85577b67-g2cf8"] Oct 30 03:56:42.156367 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:42.156283 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f733ab9-e299-4011-a75c-1b6020f2cc38" path="/var/lib/kubelet/pods/2f733ab9-e299-4011-a75c-1b6020f2cc38/volumes" Oct 30 03:56:42.586120 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:42.586081 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:56:42.586652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:42.586331 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" containerID="cri-o://ca2ccdf64b522ce8e57bccb415bdaf0823fc89197f2472166b031fbdd965c50d" gracePeriod=30 Oct 30 03:56:47.851062 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.851001 4560 generic.go:356] "Generic (PLEG): container finished" podID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerID="ca2ccdf64b522ce8e57bccb415bdaf0823fc89197f2472166b031fbdd965c50d" exitCode=0 Oct 30 03:56:47.851384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.851076 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" event={"ID":"53b97ed1-2c90-4f15-9c87-a7b5901d22dd","Type":"ContainerDied","Data":"ca2ccdf64b522ce8e57bccb415bdaf0823fc89197f2472166b031fbdd965c50d"} Oct 30 03:56:47.851384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.851143 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" event={"ID":"53b97ed1-2c90-4f15-9c87-a7b5901d22dd","Type":"ContainerDied","Data":"f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77"} Oct 30 03:56:47.851384 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.851170 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f33d533f4055b12090b1d3abfaddefd57cb63abce7c6789601d902d057bc5b77" Oct 30 03:56:47.866383 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.866356 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:56:47.994118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994077 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994175 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994220 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994280 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994315 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994299 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994359 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994377 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994387 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.994552 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994411 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994603 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.994780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994592 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\") pod \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\" (UID: \"53b97ed1-2c90-4f15-9c87-a7b5901d22dd\") " Oct 30 03:56:47.994780 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994657 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:47.994935 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.994821 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.995098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.995073 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.995698 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.995672 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.995841 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.995825 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.996006 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.995992 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.996106 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.996093 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.996224 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.996212 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.997153 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.997121 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.997243 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.997159 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "kube-api-access-dv6cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:47.998127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.998101 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:47.998426 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:47.998403 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token" pod "53b97ed1-2c90-4f15-9c87-a7b5901d22dd" (UID: "53b97ed1-2c90-4f15-9c87-a7b5901d22dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:48.097729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.097692 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6cj\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-kube-api-access-dv6cj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.097729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.097727 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.097729 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.097737 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.097974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.097746 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b97ed1-2c90-4f15-9c87-a7b5901d22dd-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.860820 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.860787 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh" Oct 30 03:56:48.928976 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.928940 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:56:48.940309 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:48.940281 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-cz5dh"] Oct 30 03:56:50.175182 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:50.175148 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" path="/var/lib/kubelet/pods/53b97ed1-2c90-4f15-9c87-a7b5901d22dd/volumes" Oct 30 03:56:52.130595 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:52.130556 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:56:57.131581 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:57.131535 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" containerID="cri-o://bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667" gracePeriod=25 Oct 30 03:56:59.167889 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:56:59.167853 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:57:04.422713 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:04.422674 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:57:04.423538 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:04.423486 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" containerID="cri-o://de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d" gracePeriod=30 Oct 30 03:57:08.447803 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:08.447759 4560 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:57:08.448208 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:08.447831 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:57:09.167324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.167284 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:57:09.709648 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.709615 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:57:09.858880 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.858839 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.858934 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.858987 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859097 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859058 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859159 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859108 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859186 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859223 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859214 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859328 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859254 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.859568 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859297 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859611 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\") pod \"9a1269bc-dd41-4506-850b-9c3afa532ca0\" (UID: \"9a1269bc-dd41-4506-850b-9c3afa532ca0\") " Oct 30 03:57:09.859696 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859648 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.859849 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.859827 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:09.860114 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.860060 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.860791 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.860763 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.861000 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.860979 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.861073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861006 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.861073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861020 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.861073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861033 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1269bc-dd41-4506-850b-9c3afa532ca0-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.861073 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861046 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.861727 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861706 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:57:09.861898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861880 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:09.861945 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.861924 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "kube-api-access-69rtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:09.862115 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.862101 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy" pod "9a1269bc-dd41-4506-850b-9c3afa532ca0" (UID: "9a1269bc-dd41-4506-850b-9c3afa532ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.962893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.962821 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.962893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.962854 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rtq\" (UniqueName: \"kubernetes.io/projected/9a1269bc-dd41-4506-850b-9c3afa532ca0-kube-api-access-69rtq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.962893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.962864 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.962893 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:09.962873 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1269bc-dd41-4506-850b-9c3afa532ca0-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:10.025469 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.025434 4560 generic.go:356] "Generic (PLEG): container finished" podID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerID="de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d" exitCode=0 Oct 30 03:57:10.025752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.025499 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" Oct 30 03:57:10.025752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.025532 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" event={"ID":"9a1269bc-dd41-4506-850b-9c3afa532ca0","Type":"ContainerDied","Data":"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d"} Oct 30 03:57:10.025752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.025627 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm" event={"ID":"9a1269bc-dd41-4506-850b-9c3afa532ca0","Type":"ContainerDied","Data":"007c1f3a559384eb058446f81cd962d9d1b9266dcabb4efb504427e219a0ae6d"} Oct 30 03:57:10.025752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.025662 4560 scope.go:122] "RemoveContainer" containerID="de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d" Oct 30 03:57:10.038004 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.037985 4560 scope.go:122] "RemoveContainer" containerID="de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d" Oct 30 03:57:10.038250 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:57:10.038233 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d\": container with ID starting with de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d not found: ID does not exist" containerID="de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d" Oct 30 03:57:10.038324 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.038258 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d"} err="failed to get container status \"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d\": rpc error: code = NotFound desc = could not find container \"de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d\": container with ID starting with de2c75334eafd4900d048e9d10870c4f819df0977fefbc8a1e0d277d0d2b798d not found: ID does not exist" Oct 30 03:57:10.081612 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.081584 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:57:10.083476 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.083447 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bqmtm"] Oct 30 03:57:10.156451 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:10.156418 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" path="/var/lib/kubelet/pods/9a1269bc-dd41-4506-850b-9c3afa532ca0/volumes" Oct 30 03:57:11.751475 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:57:11.751431 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:57:17.428023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.427989 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:57:17.463871 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.463833 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\") pod \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " Oct 30 03:57:17.464066 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.464043 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\" (UniqueName: \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\") pod \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\" (UID: \"61e1b1f5-c60c-4869-9f37-ae204e5e86a8\") " Oct 30 03:57:17.464311 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.464291 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume" pod "61e1b1f5-c60c-4869-9f37-ae204e5e86a8" (UID: "61e1b1f5-c60c-4869-9f37-ae204e5e86a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:17.464707 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.464677 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:17.475178 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.475145 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94" pod "61e1b1f5-c60c-4869-9f37-ae204e5e86a8" (UID: "61e1b1f5-c60c-4869-9f37-ae204e5e86a8"). InnerVolumeSpecName "kube-api-access-jsp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:17.565939 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:17.565898 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsp94\" (UniqueName: \"kubernetes.io/projected/61e1b1f5-c60c-4869-9f37-ae204e5e86a8-kube-api-access-jsp94\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:57:18.086259 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.086222 4560 generic.go:356] "Generic (PLEG): container finished" podID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerID="bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667" exitCode=0 Oct 30 03:57:18.086460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.086265 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" Oct 30 03:57:18.086460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.086302 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" event={"ID":"61e1b1f5-c60c-4869-9f37-ae204e5e86a8","Type":"ContainerDied","Data":"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667"} Oct 30 03:57:18.086460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.086356 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl" event={"ID":"61e1b1f5-c60c-4869-9f37-ae204e5e86a8","Type":"ContainerDied","Data":"67d8beae25986cc1b2a5684e1009e9c9f1684476a0f3cfd0c91d83037a3751ec"} Oct 30 03:57:18.086460 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.086386 4560 scope.go:122] "RemoveContainer" containerID="bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667" Oct 30 03:57:18.097344 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.097324 4560 scope.go:122] "RemoveContainer" containerID="bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667" Oct 30 03:57:18.097656 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:57:18.097631 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667\": container with ID starting with bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667 not found: ID does not exist" containerID="bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667" Oct 30 03:57:18.097720 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.097666 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667"} err="failed to get container status \"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667\": rpc error: code = NotFound desc = could not find container \"bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667\": container with ID starting with bbccc0dc565b4d33a8f51e06780a49c763f2c6552428e7b27adbd47368104667 not found: ID does not exist" Oct 30 03:57:18.139641 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.139607 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:57:18.153984 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:18.153956 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35fc87-787f7d88f9-dcrzl"] Oct 30 03:57:20.161150 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:20.161106 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" path="/var/lib/kubelet/pods/61e1b1f5-c60c-4869-9f37-ae204e5e86a8/volumes" Oct 30 03:57:41.751055 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:57:41.750972 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:57:53.230440 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:57:53.230405 4560 scope.go:122] "RemoveContainer" containerID="ca2ccdf64b522ce8e57bccb415bdaf0823fc89197f2472166b031fbdd965c50d" Oct 30 03:58:11.751705 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:58:11.751663 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:58:37.811634 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:37.811594 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:58:41.827278 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:58:41.827230 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:58:47.736752 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:47.736714 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:58:47.737163 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:47.736936 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" containerID="cri-o://ee07daeb9627703104bb9b9d75886adff480b48a6fc86b95a0d1014f7f195d24" gracePeriod=30 Oct 30 03:58:48.906127 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:48.906086 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:58:48.906506 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:48.906183 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:58:49.053638 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:49.053599 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:58:49.053856 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:49.053833 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" containerID="cri-o://51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d" gracePeriod=30 Oct 30 03:58:52.850669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:52.850628 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" podUID="ebf8aa52-21af-4771-b42d-ee5009304f72" containerName="main" containerID="cri-o://5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02" gracePeriod=45 Oct 30 03:58:52.892669 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:52.892636 4560 generic.go:356] "Generic (PLEG): container finished" podID="671fed3c-4bdf-442e-968f-030678e14961" containerID="ee07daeb9627703104bb9b9d75886adff480b48a6fc86b95a0d1014f7f195d24" exitCode=0 Oct 30 03:58:52.892874 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:52.892707 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" event={"ID":"671fed3c-4bdf-442e-968f-030678e14961","Type":"ContainerDied","Data":"ee07daeb9627703104bb9b9d75886adff480b48a6fc86b95a0d1014f7f195d24"} Oct 30 03:58:53.052255 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.051913 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:58:53.166692 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.166665 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:58:53.184060 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184026 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184060 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184059 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184111 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184159 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184230 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184291 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184287 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184341 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.184529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184415 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184444 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184491 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\") pod \"671fed3c-4bdf-442e-968f-030678e14961\" (UID: \"671fed3c-4bdf-442e-968f-030678e14961\") " Oct 30 03:58:53.184745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184590 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.184745 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184609 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.184926 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184903 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.185014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.184920 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:53.185304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.185280 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.185304 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.185307 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.185508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.185324 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.185508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.185338 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671fed3c-4bdf-442e-968f-030678e14961-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.185508 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.185350 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.186651 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.186627 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:58:53.186917 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.186892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.187009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.186973 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:53.187009 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.187000 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw" pod "671fed3c-4bdf-442e-968f-030678e14961" (UID: "671fed3c-4bdf-442e-968f-030678e14961"). InnerVolumeSpecName "kube-api-access-pb7kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:53.268733 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.268653 4560 scope.go:122] "RemoveContainer" containerID="ee07daeb9627703104bb9b9d75886adff480b48a6fc86b95a0d1014f7f195d24" Oct 30 03:58:53.285941 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.285916 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\" (UniqueName: \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.285999 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286071 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286035 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286192 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286093 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286256 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286239 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286269 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\") pod \"ebf8aa52-21af-4771-b42d-ee5009304f72\" (UID: \"ebf8aa52-21af-4771-b42d-ee5009304f72\") " Oct 30 03:58:53.286312 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286272 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.286394 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286325 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.286631 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286603 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.287014 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.286991 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287019 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671fed3c-4bdf-442e-968f-030678e14961-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287035 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671fed3c-4bdf-442e-968f-030678e14961-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287050 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7kw\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-kube-api-access-pb7kw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287065 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287078 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.287118 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.287089 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671fed3c-4bdf-442e-968f-030678e14961-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.288488 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.288461 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:53.288639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.288462 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "kube-api-access-2d4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:53.288639 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.288572 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs" pod "ebf8aa52-21af-4771-b42d-ee5009304f72" (UID: "ebf8aa52-21af-4771-b42d-ee5009304f72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:58:53.391002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.388678 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8aa52-21af-4771-b42d-ee5009304f72-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.391002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.388728 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4bh\" (UniqueName: \"kubernetes.io/projected/ebf8aa52-21af-4771-b42d-ee5009304f72-kube-api-access-2d4bh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.391002 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.388742 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8aa52-21af-4771-b42d-ee5009304f72-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:53.901144 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901101 4560 generic.go:356] "Generic (PLEG): container finished" podID="ebf8aa52-21af-4771-b42d-ee5009304f72" containerID="5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02" exitCode=0 Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901162 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901200 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" event={"ID":"ebf8aa52-21af-4771-b42d-ee5009304f72","Type":"ContainerDied","Data":"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02"} Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901258 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj" event={"ID":"ebf8aa52-21af-4771-b42d-ee5009304f72","Type":"ContainerDied","Data":"46ab64b5b17e8d12a92878181b4bb64cf5b96c7c6cc967a5507c169a3891464f"} Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901285 4560 scope.go:122] "RemoveContainer" containerID="5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02" Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901397 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" event={"ID":"671fed3c-4bdf-442e-968f-030678e14961","Type":"ContainerDied","Data":"bfd02a4973a911178b1d8fb31178d9d8e591556c9bbb1dba200fae95bc34166b"} Oct 30 03:58:53.901672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.901488 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k" Oct 30 03:58:53.911613 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.911597 4560 scope.go:122] "RemoveContainer" containerID="5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02" Oct 30 03:58:53.912023 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:58:53.911882 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02\": container with ID starting with 5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02 not found: ID does not exist" containerID="5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02" Oct 30 03:58:53.912023 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.911906 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02"} err="failed to get container status \"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02\": rpc error: code = NotFound desc = could not find container \"5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02\": container with ID starting with 5029947b78d970277df98b1916d13249a93603994168f7a1642de867a93bdc02 not found: ID does not exist" Oct 30 03:58:53.958715 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.958684 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:58:53.971316 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:53.971286 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3cfed7/test-model-3cfed7-kserve-5b6bf6f44c-c4hlj"] Oct 30 03:58:54.031947 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.031870 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:58:54.036744 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.036662 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-l4k9k"] Oct 30 03:58:54.154409 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.154377 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671fed3c-4bdf-442e-968f-030678e14961" path="/var/lib/kubelet/pods/671fed3c-4bdf-442e-968f-030678e14961/volumes" Oct 30 03:58:54.155078 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.155059 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf8aa52-21af-4771-b42d-ee5009304f72" path="/var/lib/kubelet/pods/ebf8aa52-21af-4771-b42d-ee5009304f72/volumes" Oct 30 03:58:54.338104 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.338077 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:58:54.502657 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502623 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.502835 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502791 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.502878 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502839 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.502974 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502955 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502992 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503057 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.502955 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:54.503091 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503061 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503115 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503161 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503132 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:54.503209 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503143 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503310 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503283 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\") pod \"88176770-8a5c-4f52-9928-2d1b0de1386f\" (UID: \"88176770-8a5c-4f52-9928-2d1b0de1386f\") " Oct 30 03:58:54.503388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503305 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:54.503388 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503336 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:54.503529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.503438 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:54.504130 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.504099 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.504263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.504133 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.504263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.504149 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88176770-8a5c-4f52-9928-2d1b0de1386f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.504263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.504164 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.504263 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.504178 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.505266 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.505235 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:54.505392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.505352 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:54.505392 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.505379 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:58:54.505479 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.505416 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl" pod "88176770-8a5c-4f52-9928-2d1b0de1386f" (UID: "88176770-8a5c-4f52-9928-2d1b0de1386f"). InnerVolumeSpecName "kube-api-access-k9cgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:54.594591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.594257 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:58:54.605824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.605779 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.605824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.605826 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.605824 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.605840 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88176770-8a5c-4f52-9928-2d1b0de1386f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.606109 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.605853 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cgl\" (UniqueName: \"kubernetes.io/projected/88176770-8a5c-4f52-9928-2d1b0de1386f-kube-api-access-k9cgl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:58:54.910317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.910238 4560 generic.go:356] "Generic (PLEG): container finished" podID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerID="51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d" exitCode=0 Oct 30 03:58:54.910317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.910290 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" Oct 30 03:58:54.910317 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.910310 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" event={"ID":"88176770-8a5c-4f52-9928-2d1b0de1386f","Type":"ContainerDied","Data":"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d"} Oct 30 03:58:54.910901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.910361 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz" event={"ID":"88176770-8a5c-4f52-9928-2d1b0de1386f","Type":"ContainerDied","Data":"2692d3b2a0328182c0ce5f289665ba35ed993e0109f036dd45188ebb0d2c311d"} Oct 30 03:58:54.910901 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.910390 4560 scope.go:122] "RemoveContainer" containerID="51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d" Oct 30 03:58:54.923591 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.923568 4560 scope.go:122] "RemoveContainer" containerID="51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d" Oct 30 03:58:54.923935 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:58:54.923910 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d\": container with ID starting with 51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d not found: ID does not exist" containerID="51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d" Oct 30 03:58:54.924012 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.923944 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d"} err="failed to get container status \"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d\": rpc error: code = NotFound desc = could not find container \"51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d\": container with ID starting with 51025c1c44adefaa07eb5b0259271a81e0a08c1f3faf705706d5fa5572eed85d not found: ID does not exist" Oct 30 03:58:54.964005 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.963972 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:58:54.976738 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:54.976707 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-p5kkz"] Oct 30 03:58:56.155898 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:58:56.155823 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" path="/var/lib/kubelet/pods/88176770-8a5c-4f52-9928-2d1b0de1386f/volumes" Oct 30 03:59:09.617805 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:09.617693 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerName="main" containerID="cri-o://3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06" gracePeriod=45 Oct 30 03:59:09.621989 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:09.621944 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.172:8000/ready\": EOF" Oct 30 03:59:09.903039 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:09.903006 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:59:10.018587 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.018552 4560 generic.go:356] "Generic (PLEG): container finished" podID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerID="3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06" exitCode=0 Oct 30 03:59:10.018770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.018595 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" Oct 30 03:59:10.018770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.018622 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" event={"ID":"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa","Type":"ContainerDied","Data":"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06"} Oct 30 03:59:10.018770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.018677 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq" event={"ID":"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa","Type":"ContainerDied","Data":"9541f1a1d528cd8c01b555bd312b8e68db4724901baad945cadbb1d323047a45"} Oct 30 03:59:10.018770 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.018694 4560 scope.go:122] "RemoveContainer" containerID="3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06" Oct 30 03:59:10.029279 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.029258 4560 scope.go:122] "RemoveContainer" containerID="3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06" Oct 30 03:59:10.029581 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:59:10.029561 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06\": container with ID starting with 3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06 not found: ID does not exist" containerID="3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06" Oct 30 03:59:10.029655 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.029592 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06"} err="failed to get container status \"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06\": rpc error: code = NotFound desc = could not find container \"3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06\": container with ID starting with 3b69760c313574e25f6d77b4b05352a9badb52d808b47d22f1f376963a41aa06 not found: ID does not exist" Oct 30 03:59:10.048814 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.048780 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.048909 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049011 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.048966 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049128 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049059 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049184 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049145 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049297 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049248 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:59:10.049423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049315 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:59:10.049423 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049321 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\" (UniqueName: \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\") pod \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\" (UID: \"947bccfd-189e-4b6b-bd92-d5a2d5fda8aa\") " Oct 30 03:59:10.049529 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049425 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:59:10.049955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049932 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.049955 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049955 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.050098 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.049968 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.051204 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.051181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:59:10.051281 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.051229 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:59:10.051343 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.051326 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5" pod "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" (UID: "947bccfd-189e-4b6b-bd92-d5a2d5fda8aa"). InnerVolumeSpecName "kube-api-access-jlnc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:59:10.151436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.151346 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.151436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.151373 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnc5\" (UniqueName: \"kubernetes.io/projected/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-kube-api-access-jlnc5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.151436 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.151383 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:10.367629 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.367589 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:59:10.377599 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:10.377560 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9e0d09/test-model-9e0d09-kserve-bcb9f9db9-6p6cq"] Oct 30 03:59:11.750717 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:59:11.750681 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:59:12.157833 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:12.157751 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" path="/var/lib/kubelet/pods/947bccfd-189e-4b6b-bd92-d5a2d5fda8aa/volumes" Oct 30 03:59:25.948861 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:25.948820 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:59:25.949454 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:25.949030 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" podUID="6089fe6e-613b-4c04-9926-175201005702" containerName="maas-api" containerID="cri-o://b15422e171d063af892e0c6c02b29de348080af54e42b41e392b577b7176b731" gracePeriod=30 Oct 30 03:59:27.158186 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.158152 4560 generic.go:356] "Generic (PLEG): container finished" podID="6089fe6e-613b-4c04-9926-175201005702" containerID="b15422e171d063af892e0c6c02b29de348080af54e42b41e392b577b7176b731" exitCode=0 Oct 30 03:59:27.158609 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.158215 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" event={"ID":"6089fe6e-613b-4c04-9926-175201005702","Type":"ContainerDied","Data":"b15422e171d063af892e0c6c02b29de348080af54e42b41e392b577b7176b731"} Oct 30 03:59:27.223307 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.223280 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:59:27.311771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.311677 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\") pod \"6089fe6e-613b-4c04-9926-175201005702\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " Oct 30 03:59:27.311771 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.311719 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\") pod \"6089fe6e-613b-4c04-9926-175201005702\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " Oct 30 03:59:27.312033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.311784 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\" (UniqueName: \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\") pod \"6089fe6e-613b-4c04-9926-175201005702\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " Oct 30 03:59:27.312033 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.311840 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") pod \"6089fe6e-613b-4c04-9926-175201005702\" (UID: \"6089fe6e-613b-4c04-9926-175201005702\") " Oct 30 03:59:27.312359 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.312327 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca" pod "6089fe6e-613b-4c04-9926-175201005702" (UID: "6089fe6e-613b-4c04-9926-175201005702"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:59:27.313652 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.313291 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6089fe6e-613b-4c04-9926-175201005702-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:27.314826 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.314798 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs" pod "6089fe6e-613b-4c04-9926-175201005702" (UID: "6089fe6e-613b-4c04-9926-175201005702"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:59:27.315215 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.315187 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls" pod "6089fe6e-613b-4c04-9926-175201005702" (UID: "6089fe6e-613b-4c04-9926-175201005702"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:59:27.315313 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.315201 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw" pod "6089fe6e-613b-4c04-9926-175201005702" (UID: "6089fe6e-613b-4c04-9926-175201005702"). InnerVolumeSpecName "kube-api-access-52snw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:59:27.415095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.415052 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:27.415095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.415086 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52snw\" (UniqueName: \"kubernetes.io/projected/6089fe6e-613b-4c04-9926-175201005702-kube-api-access-52snw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:27.415095 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:27.415095 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6089fe6e-613b-4c04-9926-175201005702-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 03:59:28.165894 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:28.165863 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" Oct 30 03:59:28.166261 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:28.165893 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm" event={"ID":"6089fe6e-613b-4c04-9926-175201005702","Type":"ContainerDied","Data":"e5d37984147bc5a9b36dcef25a9cf8a015fbb99be83e8d5c65f436e307b29866"} Oct 30 03:59:28.166261 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:28.165950 4560 scope.go:122] "RemoveContainer" containerID="b15422e171d063af892e0c6c02b29de348080af54e42b41e392b577b7176b731" Oct 30 03:59:28.219484 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:28.219448 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:59:28.232167 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:28.232134 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cfed7-7d7d56b8f9-mckrm"] Oct 30 03:59:30.153834 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:30.153801 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6089fe6e-613b-4c04-9926-175201005702" path="/var/lib/kubelet/pods/6089fe6e-613b-4c04-9926-175201005702/volumes" Oct 30 03:59:39.963672 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:39.963631 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 03:59:41.750591 ip-10-0-136-97 kubenswrapper[4560]: E1030 03:59:41.750544 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 03:59:44.964845 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:44.964805 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" containerID="cri-o://5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682" gracePeriod=25 Oct 30 03:59:48.468617 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:48.468559 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:59:49.792142 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:49.792107 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 03:59:58.468849 ip-10-0-136-97 kubenswrapper[4560]: I1030 03:59:58.468803 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:00:00.245473 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.245433 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:00.246954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246923 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" Oct 30 04:00:00.246954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246952 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246968 4560 cpu_manager.go:414] "containerMap: removing container" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246976 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246992 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c38db9cc-d6da-40bb-925b-32a44080ad50" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.246999 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38db9cc-d6da-40bb-925b-32a44080ad50" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247024 4560 cpu_manager.go:414] "containerMap: removing container" podUID="65719913-37b3-45ee-8eef-ef409648552c" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247031 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65719913-37b3-45ee-8eef-ef409648552c" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247049 4560 cpu_manager.go:414] "containerMap: removing container" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247056 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247071 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6089fe6e-613b-4c04-9926-175201005702" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247077 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6089fe6e-613b-4c04-9926-175201005702" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247090 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" containerName="main" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247096 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" containerName="main" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247109 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247116 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247129 4560 cpu_manager.go:414] "containerMap: removing container" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerName="main" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247136 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerName="main" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247152 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2f733ab9-e299-4011-a75c-1b6020f2cc38" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247159 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f733ab9-e299-4011-a75c-1b6020f2cc38" containerName="maas-api" Oct 30 04:00:00.247174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247175 4560 cpu_manager.go:414] "containerMap: removing container" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247182 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247195 4560 cpu_manager.go:414] "containerMap: removing container" podUID="576b81e6-4ed6-4e25-806a-a09405ba4679" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247203 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576b81e6-4ed6-4e25-806a-a09405ba4679" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247217 4560 cpu_manager.go:414] "containerMap: removing container" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247224 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247239 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ebf8aa52-21af-4771-b42d-ee5009304f72" containerName="main" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247245 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf8aa52-21af-4771-b42d-ee5009304f72" containerName="main" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247269 4560 cpu_manager.go:414] "containerMap: removing container" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247277 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247298 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247306 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247465 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38db9cc-d6da-40bb-925b-32a44080ad50" containerName="maas-api" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247490 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="65719913-37b3-45ee-8eef-ef409648552c" containerName="maas-api" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247506 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="006179d2-d715-4f91-a12e-7944d0842361" containerName="payload-pre-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247532 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6089fe6e-613b-4c04-9926-175201005702" containerName="maas-api" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247543 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a26b4c2-d0da-4cf5-a838-a08d72f48ff2" containerName="main" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247553 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0326dcb-0dcc-49f7-a46f-3a1611e438bd" containerName="payload-pre-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247564 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="947bccfd-189e-4b6b-bd92-d5a2d5fda8aa" containerName="main" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247575 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f733ab9-e299-4011-a75c-1b6020f2cc38" containerName="maas-api" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247587 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e1b1f5-c60c-4869-9f37-ae204e5e86a8" containerName="payload-pre-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247595 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="576b81e6-4ed6-4e25-806a-a09405ba4679" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247605 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="88176770-8a5c-4f52-9928-2d1b0de1386f" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247615 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf8aa52-21af-4771-b42d-ee5009304f72" containerName="main" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247626 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="671fed3c-4bdf-442e-968f-030678e14961" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247638 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c267219-da81-4b92-8015-f49f66e2dd08" containerName="payload-processing" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247649 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1269bc-dd41-4506-850b-9c3afa532ca0" containerName="istio-proxy" Oct 30 04:00:00.248126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.247658 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b97ed1-2c90-4f15-9c87-a7b5901d22dd" containerName="istio-proxy" Oct 30 04:00:00.254185 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.253103 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.254725 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.254697 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:00.285852 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.285818 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:00.286719 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.286129 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.292303 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.292276 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:00.292558 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.292497 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:00.292794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.292777 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.303313 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.303276 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:00.371890 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.371854 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 04:00:00.418927 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.418889 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.419100 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.419025 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vl26\" (UniqueName: \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.419100 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.419074 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.419176 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.419104 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bt26\" (UniqueName: \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.419229 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.419212 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.419273 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.419262 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whlk\" (UniqueName: \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.520361 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520274 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.520361 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520346 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vl26\" (UniqueName: \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.520659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520405 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.520659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520447 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bt26\" (UniqueName: \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.520659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520506 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.520659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.520606 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whlk\" (UniqueName: \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.521462 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.521145 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.521462 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.521260 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.521462 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.521379 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.540060 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.540036 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bt26\" (UniqueName: \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\") pod \"maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.540193 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.540169 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vl26\" (UniqueName: \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\") pod \"maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.540419 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.540402 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whlk\" (UniqueName: \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.615629 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.615587 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:00.634789 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.634761 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:00.640137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.640112 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:00.939104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.939076 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:00.939674 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:00:00.939649 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03893764_1fb6_43de_9972_917cc084c474.slice/crio-7f25b2ee65821735d823251ad3b754080d01d30fdfa13a1d4a73639be3be3bf0 WatchSource:0}: Error finding container 7f25b2ee65821735d823251ad3b754080d01d30fdfa13a1d4a73639be3be3bf0: Status 404 returned error can't find the container with id 7f25b2ee65821735d823251ad3b754080d01d30fdfa13a1d4a73639be3be3bf0 Oct 30 04:00:00.976147 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:00:00.976074 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edc8fb6_bd2d_4e0e_bdb9_8659cf5330d8.slice/crio-20a39c227a4e2d8b17fc3384830f33126993686dfcc3b325a54517caed72bf8e WatchSource:0}: Error finding container 20a39c227a4e2d8b17fc3384830f33126993686dfcc3b325a54517caed72bf8e: Status 404 returned error can't find the container with id 20a39c227a4e2d8b17fc3384830f33126993686dfcc3b325a54517caed72bf8e Oct 30 04:00:00.976346 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.976287 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:00.978628 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:00:00.978598 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304dac9a_5763_475a_9107_a12d703cb8f0.slice/crio-782647712d4522d49e5ff4a94a98abbb7f1af3228b53ebf14c8169bfefc91955 WatchSource:0}: Error finding container 782647712d4522d49e5ff4a94a98abbb7f1af3228b53ebf14c8169bfefc91955: Status 404 returned error can't find the container with id 782647712d4522d49e5ff4a94a98abbb7f1af3228b53ebf14c8169bfefc91955 Oct 30 04:00:00.980295 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:00.980270 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:01.409155 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:01.409116 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerStarted","Data":"20a39c227a4e2d8b17fc3384830f33126993686dfcc3b325a54517caed72bf8e"} Oct 30 04:00:01.410812 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:01.410783 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerStarted","Data":"224314e9c5d87bd9617c668168a648cf7f7d91c5749761e32ba4933093132ec7"} Oct 30 04:00:01.410954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:01.410829 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerStarted","Data":"782647712d4522d49e5ff4a94a98abbb7f1af3228b53ebf14c8169bfefc91955"} Oct 30 04:00:01.412161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:01.412135 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerStarted","Data":"7f25b2ee65821735d823251ad3b754080d01d30fdfa13a1d4a73639be3be3bf0"} Oct 30 04:00:01.455580 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:01.455476 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" podStartSLOduration=1.455448949 podStartE2EDuration="1.455448949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:01.450730689 +0000 UTC m=+3429.435850578" watchObservedRunningTime="2026-10-30 04:00:01.455448949 +0000 UTC m=+3429.440568844" Oct 30 04:00:02.421484 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.421397 4560 generic.go:356] "Generic (PLEG): container finished" podID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerID="3a9af04cfdb4333ec61a3c0b9ead6584b58d014599a673e3279294abfbb485a0" exitCode=60 Oct 30 04:00:02.421963 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.421482 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerDied","Data":"3a9af04cfdb4333ec61a3c0b9ead6584b58d014599a673e3279294abfbb485a0"} Oct 30 04:00:02.421963 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.421859 4560 scope.go:122] "RemoveContainer" containerID="3a9af04cfdb4333ec61a3c0b9ead6584b58d014599a673e3279294abfbb485a0" Oct 30 04:00:02.423315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.423288 4560 generic.go:356] "Generic (PLEG): container finished" podID="304dac9a-5763-475a-9107-a12d703cb8f0" containerID="224314e9c5d87bd9617c668168a648cf7f7d91c5749761e32ba4933093132ec7" exitCode=60 Oct 30 04:00:02.423315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.423318 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerDied","Data":"224314e9c5d87bd9617c668168a648cf7f7d91c5749761e32ba4933093132ec7"} Oct 30 04:00:02.423663 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.423643 4560 scope.go:122] "RemoveContainer" containerID="224314e9c5d87bd9617c668168a648cf7f7d91c5749761e32ba4933093132ec7" Oct 30 04:00:02.425338 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.425258 4560 generic.go:356] "Generic (PLEG): container finished" podID="03893764-1fb6-43de-9972-917cc084c474" containerID="91fec223d73058a47c8f1f4099be3d3a41c0d61ce5e8b2dffe1c379cb22786cc" exitCode=60 Oct 30 04:00:02.425338 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.425315 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerDied","Data":"91fec223d73058a47c8f1f4099be3d3a41c0d61ce5e8b2dffe1c379cb22786cc"} Oct 30 04:00:02.425710 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:02.425691 4560 scope.go:122] "RemoveContainer" containerID="91fec223d73058a47c8f1f4099be3d3a41c0d61ce5e8b2dffe1c379cb22786cc" Oct 30 04:00:03.434368 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:03.434328 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerStarted","Data":"48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7"} Oct 30 04:00:03.436467 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:03.436442 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerStarted","Data":"25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6"} Oct 30 04:00:03.438261 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:03.438240 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerStarted","Data":"541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4"} Oct 30 04:00:03.476320 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:03.476264 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" podStartSLOduration=3.476247398 podStartE2EDuration="3.476247398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.475850213 +0000 UTC m=+3431.460970112" watchObservedRunningTime="2026-10-30 04:00:03.476247398 +0000 UTC m=+3431.461367287" Oct 30 04:00:03.517264 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:03.517206 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" podStartSLOduration=3.517191478 podStartE2EDuration="3.517191478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.51473538 +0000 UTC m=+3431.499855291" watchObservedRunningTime="2026-10-30 04:00:03.517191478 +0000 UTC m=+3431.502311440" Oct 30 04:00:04.447275 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.447235 4560 generic.go:356] "Generic (PLEG): container finished" podID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerID="25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6" exitCode=60 Oct 30 04:00:04.447770 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.447316 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerDied","Data":"25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6"} Oct 30 04:00:04.447770 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.447390 4560 scope.go:122] "RemoveContainer" containerID="3a9af04cfdb4333ec61a3c0b9ead6584b58d014599a673e3279294abfbb485a0" Oct 30 04:00:04.447770 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.447767 4560 scope.go:122] "RemoveContainer" containerID="25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6" Oct 30 04:00:04.448162 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:04.448137 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr_odh-ai-gateway-infra(8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" Oct 30 04:00:04.449429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.449405 4560 generic.go:356] "Generic (PLEG): container finished" podID="304dac9a-5763-475a-9107-a12d703cb8f0" containerID="541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4" exitCode=60 Oct 30 04:00:04.449575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.449492 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerDied","Data":"541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4"} Oct 30 04:00:04.449795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.449773 4560 scope.go:122] "RemoveContainer" containerID="541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4" Oct 30 04:00:04.450296 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:04.450272 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm_odh-ai-gateway-infra(304dac9a-5763-475a-9107-a12d703cb8f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" Oct 30 04:00:04.453757 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.453734 4560 generic.go:356] "Generic (PLEG): container finished" podID="03893764-1fb6-43de-9972-917cc084c474" containerID="48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7" exitCode=60 Oct 30 04:00:04.454000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.453981 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerDied","Data":"48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7"} Oct 30 04:00:04.454368 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.454350 4560 scope.go:122] "RemoveContainer" containerID="48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7" Oct 30 04:00:04.454701 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:04.454673 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7_odh-ai-gateway-infra(03893764-1fb6-43de-9972-917cc084c474)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" podUID="03893764-1fb6-43de-9972-917cc084c474" Oct 30 04:00:04.461710 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.461694 4560 scope.go:122] "RemoveContainer" containerID="224314e9c5d87bd9617c668168a648cf7f7d91c5749761e32ba4933093132ec7" Oct 30 04:00:04.474792 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.474773 4560 scope.go:122] "RemoveContainer" containerID="91fec223d73058a47c8f1f4099be3d3a41c0d61ce5e8b2dffe1c379cb22786cc" Oct 30 04:00:04.815998 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:04.815955 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" podUID="055098bc-6674-4e59-8619-afae8f20d297" containerName="main" containerID="cri-o://df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3" gracePeriod=45 Oct 30 04:00:05.117101 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.117073 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 04:00:05.191921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.191883 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\" (UniqueName: \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.192113 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.192041 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.192113 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.192091 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\" (UniqueName: \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.192238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.192146 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.192238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.192186 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.192355 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.192246 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\") pod \"055098bc-6674-4e59-8619-afae8f20d297\" (UID: \"055098bc-6674-4e59-8619-afae8f20d297\") " Oct 30 04:00:05.193191 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.193147 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:05.193557 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.193508 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.194133 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.194106 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:05.194430 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.194289 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:05.197053 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.197015 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:05.197690 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.197659 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "kube-api-access-6m9pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:05.198900 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.198877 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm" pod "055098bc-6674-4e59-8619-afae8f20d297" (UID: "055098bc-6674-4e59-8619-afae8f20d297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:05.228842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.228815 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 04:00:05.294493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.294449 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\" (UniqueName: \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\") pod \"636fe6a3-c210-4ad2-8f64-80a61125d145\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " Oct 30 04:00:05.294704 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.294639 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\") pod \"636fe6a3-c210-4ad2-8f64-80a61125d145\" (UID: \"636fe6a3-c210-4ad2-8f64-80a61125d145\") " Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295352 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295383 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295399 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/055098bc-6674-4e59-8619-afae8f20d297-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295399 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume" pod "636fe6a3-c210-4ad2-8f64-80a61125d145" (UID: "636fe6a3-c210-4ad2-8f64-80a61125d145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295411 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/055098bc-6674-4e59-8619-afae8f20d297-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.295599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.295456 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9pt\" (UniqueName: \"kubernetes.io/projected/055098bc-6674-4e59-8619-afae8f20d297-kube-api-access-6m9pt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.304544 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.297427 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf" pod "636fe6a3-c210-4ad2-8f64-80a61125d145" (UID: "636fe6a3-c210-4ad2-8f64-80a61125d145"). InnerVolumeSpecName "kube-api-access-rvxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:05.401206 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.401101 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxwf\" (UniqueName: \"kubernetes.io/projected/636fe6a3-c210-4ad2-8f64-80a61125d145-kube-api-access-rvxwf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.401206 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.401138 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636fe6a3-c210-4ad2-8f64-80a61125d145-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:05.462924 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.462890 4560 scope.go:122] "RemoveContainer" containerID="25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6" Oct 30 04:00:05.463414 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:05.463251 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr_odh-ai-gateway-infra(8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" Oct 30 04:00:05.464670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.464643 4560 scope.go:122] "RemoveContainer" containerID="541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4" Oct 30 04:00:05.464962 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:05.464944 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm_odh-ai-gateway-infra(304dac9a-5763-475a-9107-a12d703cb8f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" Oct 30 04:00:05.466286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.466257 4560 scope.go:122] "RemoveContainer" containerID="48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7" Oct 30 04:00:05.466646 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:05.466612 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7_odh-ai-gateway-infra(03893764-1fb6-43de-9972-917cc084c474)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" podUID="03893764-1fb6-43de-9972-917cc084c474" Oct 30 04:00:05.467594 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.467569 4560 generic.go:356] "Generic (PLEG): container finished" podID="055098bc-6674-4e59-8619-afae8f20d297" containerID="df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3" exitCode=0 Oct 30 04:00:05.467694 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.467632 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" Oct 30 04:00:05.467694 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.467653 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" event={"ID":"055098bc-6674-4e59-8619-afae8f20d297","Type":"ContainerDied","Data":"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3"} Oct 30 04:00:05.467809 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.467695 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k" event={"ID":"055098bc-6674-4e59-8619-afae8f20d297","Type":"ContainerDied","Data":"04bda07c8406243f0ace140aba025f8d5597c4aa6d6f4dab5842ce8d31f9640d"} Oct 30 04:00:05.467809 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.467716 4560 scope.go:122] "RemoveContainer" containerID="df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3" Oct 30 04:00:05.469209 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.469183 4560 generic.go:356] "Generic (PLEG): container finished" podID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerID="5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682" exitCode=0 Oct 30 04:00:05.469287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.469220 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" event={"ID":"636fe6a3-c210-4ad2-8f64-80a61125d145","Type":"ContainerDied","Data":"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682"} Oct 30 04:00:05.469339 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.469242 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" Oct 30 04:00:05.469429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.469324 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5" event={"ID":"636fe6a3-c210-4ad2-8f64-80a61125d145","Type":"ContainerDied","Data":"655fd84c801ddec6aa131933dc1fcf37803bcce280fa6379b4751b76024c91c6"} Oct 30 04:00:05.479011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.478986 4560 scope.go:122] "RemoveContainer" containerID="df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3" Oct 30 04:00:05.479373 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:05.479344 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3\": container with ID starting with df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3 not found: ID does not exist" containerID="df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3" Oct 30 04:00:05.479456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.479389 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3"} err="failed to get container status \"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3\": rpc error: code = NotFound desc = could not find container \"df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3\": container with ID starting with df46e1f80afcac5ac58a3b783013972ba119094c1f81f6ac3e42196567ff79a3 not found: ID does not exist" Oct 30 04:00:05.479456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.479413 4560 scope.go:122] "RemoveContainer" containerID="5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682" Oct 30 04:00:05.489416 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.489394 4560 scope.go:122] "RemoveContainer" containerID="5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682" Oct 30 04:00:05.490545 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:05.490489 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682\": container with ID starting with 5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682 not found: ID does not exist" containerID="5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682" Oct 30 04:00:05.490637 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.490558 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682"} err="failed to get container status \"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682\": rpc error: code = NotFound desc = could not find container \"5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682\": container with ID starting with 5e0c40481ca4da751b487f1cf3ea72307d6468aeb9f90b351953dea953903682 not found: ID does not exist" Oct 30 04:00:05.537457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.537422 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 04:00:05.544222 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.544189 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e0d09-bc9b4cb4c-kwzp5"] Oct 30 04:00:05.576472 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.576443 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 04:00:05.582231 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:05.582202 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a3d585/test-model-a3d585-kserve-845b794975-j897k"] Oct 30 04:00:06.154827 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:06.154741 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055098bc-6674-4e59-8619-afae8f20d297" path="/var/lib/kubelet/pods/055098bc-6674-4e59-8619-afae8f20d297/volumes" Oct 30 04:00:06.155303 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:06.155286 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" path="/var/lib/kubelet/pods/636fe6a3-c210-4ad2-8f64-80a61125d145/volumes" Oct 30 04:00:07.014441 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:07.014410 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 04:00:11.367125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:11.367087 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 04:00:11.367619 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:11.367293 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" containerID="cri-o://628219daa7e2216b097eb5aa841b55065651bc32911611342718c365f40effc0" gracePeriod=30 Oct 30 04:00:11.751510 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:11.751476 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:00:14.145478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:14.145443 4560 scope.go:122] "RemoveContainer" containerID="541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4" Oct 30 04:00:14.535577 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:14.535509 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerStarted","Data":"f6de60c92d5a83f37e533540b8ffc097c3617fa8ef986252c61a62d4c278aa76"} Oct 30 04:00:15.145029 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.144996 4560 scope.go:122] "RemoveContainer" containerID="25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6" Oct 30 04:00:15.544481 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.544437 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerStarted","Data":"a62e1af04d8ff3da79ac8a72e77cb4a913df702f4cffdb74afb0d92b0d494a42"} Oct 30 04:00:15.546390 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.546362 4560 generic.go:356] "Generic (PLEG): container finished" podID="304dac9a-5763-475a-9107-a12d703cb8f0" containerID="f6de60c92d5a83f37e533540b8ffc097c3617fa8ef986252c61a62d4c278aa76" exitCode=60 Oct 30 04:00:15.546553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.546429 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerDied","Data":"f6de60c92d5a83f37e533540b8ffc097c3617fa8ef986252c61a62d4c278aa76"} Oct 30 04:00:15.546553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.546485 4560 scope.go:122] "RemoveContainer" containerID="541e8c796279131940bd0deace82ceb8ba3b8272b19fe45dd27ecb6a6a039bd4" Oct 30 04:00:15.546843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.546825 4560 scope.go:122] "RemoveContainer" containerID="f6de60c92d5a83f37e533540b8ffc097c3617fa8ef986252c61a62d4c278aa76" Oct 30 04:00:15.547166 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:15.547142 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm_odh-ai-gateway-infra(304dac9a-5763-475a-9107-a12d703cb8f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" Oct 30 04:00:15.583621 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:15.583595 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:16.559171 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.559137 4560 generic.go:356] "Generic (PLEG): container finished" podID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerID="a62e1af04d8ff3da79ac8a72e77cb4a913df702f4cffdb74afb0d92b0d494a42" exitCode=60 Oct 30 04:00:16.559670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.559223 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerDied","Data":"a62e1af04d8ff3da79ac8a72e77cb4a913df702f4cffdb74afb0d92b0d494a42"} Oct 30 04:00:16.559670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.559285 4560 scope.go:122] "RemoveContainer" containerID="25e92a15a3de519a3a1538efca06c361983fa77d07d93c72db693844ce17c4e6" Oct 30 04:00:16.559781 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.559749 4560 scope.go:122] "RemoveContainer" containerID="a62e1af04d8ff3da79ac8a72e77cb4a913df702f4cffdb74afb0d92b0d494a42" Oct 30 04:00:16.560057 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:16.560033 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr_odh-ai-gateway-infra(8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" Oct 30 04:00:16.565997 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.565396 4560 generic.go:356] "Generic (PLEG): container finished" podID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerID="628219daa7e2216b097eb5aa841b55065651bc32911611342718c365f40effc0" exitCode=0 Oct 30 04:00:16.565997 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.565619 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" event={"ID":"64fddf95-b5e1-4d79-a65f-92fb3b6e1295","Type":"ContainerDied","Data":"628219daa7e2216b097eb5aa841b55065651bc32911611342718c365f40effc0"} Oct 30 04:00:16.602665 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.602611 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:16.651228 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.651198 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 04:00:16.732142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.732111 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:16.756359 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756325 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756557 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756382 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756557 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756441 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756557 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756509 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756594 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756625 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756648 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756675 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756715 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756763 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:16.756840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756822 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:16.757218 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.756904 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\") pod \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\" (UID: \"64fddf95-b5e1-4d79-a65f-92fb3b6e1295\") " Oct 30 04:00:16.757218 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757150 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:16.757362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757329 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:16.757559 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757541 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.757674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757564 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.757674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757579 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.757674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757592 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.757674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.757609 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.759142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.759120 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:16.759142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.759130 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:16.759311 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.759250 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "kube-api-access-jffzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:16.759311 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.759258 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy" pod "64fddf95-b5e1-4d79-a65f-92fb3b6e1295" (UID: "64fddf95-b5e1-4d79-a65f-92fb3b6e1295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:16.858561 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.858506 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\" (UniqueName: \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\") pod \"304dac9a-5763-475a-9107-a12d703cb8f0\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " Oct 30 04:00:16.858971 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.858946 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\") pod \"304dac9a-5763-475a-9107-a12d703cb8f0\" (UID: \"304dac9a-5763-475a-9107-a12d703cb8f0\") " Oct 30 04:00:16.859324 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859296 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca" pod "304dac9a-5763-475a-9107-a12d703cb8f0" (UID: "304dac9a-5763-475a-9107-a12d703cb8f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:16.859583 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859565 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.859671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859587 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.859671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859600 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.859671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859612 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffzp\" (UniqueName: \"kubernetes.io/projected/64fddf95-b5e1-4d79-a65f-92fb3b6e1295-kube-api-access-jffzp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.859671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.859627 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304dac9a-5763-475a-9107-a12d703cb8f0-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.860879 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.860858 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk" pod "304dac9a-5763-475a-9107-a12d703cb8f0" (UID: "304dac9a-5763-475a-9107-a12d703cb8f0"). InnerVolumeSpecName "kube-api-access-2whlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:16.960800 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.960766 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whlk\" (UniqueName: \"kubernetes.io/projected/304dac9a-5763-475a-9107-a12d703cb8f0-kube-api-access-2whlk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:16.998117 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.998051 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 04:00:16.998372 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:16.998340 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" podUID="d73ccabf-284e-4670-b12b-0df010cf5aed" containerName="maas-api" containerID="cri-o://c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f" gracePeriod=30 Oct 30 04:00:17.145720 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.145688 4560 scope.go:122] "RemoveContainer" containerID="48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7" Oct 30 04:00:17.387422 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.387392 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 04:00:17.473621 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.472295 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\") pod \"d73ccabf-284e-4670-b12b-0df010cf5aed\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " Oct 30 04:00:17.473621 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.472436 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\") pod \"d73ccabf-284e-4670-b12b-0df010cf5aed\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " Oct 30 04:00:17.473621 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.472490 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\") pod \"d73ccabf-284e-4670-b12b-0df010cf5aed\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " Oct 30 04:00:17.473621 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.472586 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\" (UniqueName: \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\") pod \"d73ccabf-284e-4670-b12b-0df010cf5aed\" (UID: \"d73ccabf-284e-4670-b12b-0df010cf5aed\") " Oct 30 04:00:17.473974 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.473687 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca" pod "d73ccabf-284e-4670-b12b-0df010cf5aed" (UID: "d73ccabf-284e-4670-b12b-0df010cf5aed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:17.477969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.477160 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs" pod "d73ccabf-284e-4670-b12b-0df010cf5aed" (UID: "d73ccabf-284e-4670-b12b-0df010cf5aed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:17.477969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.477306 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l" pod "d73ccabf-284e-4670-b12b-0df010cf5aed" (UID: "d73ccabf-284e-4670-b12b-0df010cf5aed"). InnerVolumeSpecName "kube-api-access-l9d9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:17.480654 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.480608 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls" pod "d73ccabf-284e-4670-b12b-0df010cf5aed" (UID: "d73ccabf-284e-4670-b12b-0df010cf5aed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:17.574947 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.574909 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73ccabf-284e-4670-b12b-0df010cf5aed-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:17.574947 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.574948 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:17.575505 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.574964 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9d9l\" (UniqueName: \"kubernetes.io/projected/d73ccabf-284e-4670-b12b-0df010cf5aed-kube-api-access-l9d9l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:17.575505 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.574978 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73ccabf-284e-4670-b12b-0df010cf5aed-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:17.576157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.576022 4560 generic.go:356] "Generic (PLEG): container finished" podID="d73ccabf-284e-4670-b12b-0df010cf5aed" containerID="c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f" exitCode=0 Oct 30 04:00:17.576157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.576086 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" Oct 30 04:00:17.576157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.576115 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" event={"ID":"d73ccabf-284e-4670-b12b-0df010cf5aed","Type":"ContainerDied","Data":"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f"} Oct 30 04:00:17.576388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.576206 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r" event={"ID":"d73ccabf-284e-4670-b12b-0df010cf5aed","Type":"ContainerDied","Data":"e185c09b3ec560b14e95da06b1d68452bbbc349ae891248f32cb4e8865bf490f"} Oct 30 04:00:17.576388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.576264 4560 scope.go:122] "RemoveContainer" containerID="c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f" Oct 30 04:00:17.592212 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.592189 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" Oct 30 04:00:17.594167 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.592205 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm" event={"ID":"304dac9a-5763-475a-9107-a12d703cb8f0","Type":"ContainerDied","Data":"782647712d4522d49e5ff4a94a98abbb7f1af3228b53ebf14c8169bfefc91955"} Oct 30 04:00:17.597665 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.597635 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" event={"ID":"64fddf95-b5e1-4d79-a65f-92fb3b6e1295","Type":"ContainerDied","Data":"91327118b20f8ecad8bd01eb88936e2e3c8071ca9a313a529aaec0acacefc354"} Oct 30 04:00:17.597805 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.597788 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh" Oct 30 04:00:17.622711 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.622682 4560 scope.go:122] "RemoveContainer" containerID="c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f" Oct 30 04:00:17.623084 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:17.623058 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f\": container with ID starting with c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f not found: ID does not exist" containerID="c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f" Oct 30 04:00:17.623173 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.623117 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f"} err="failed to get container status \"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f\": rpc error: code = NotFound desc = could not find container \"c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f\": container with ID starting with c49d71f8475c99102515ee1551df32bac99a1b9ec3906e5352e1fad40c68588f not found: ID does not exist" Oct 30 04:00:17.623173 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.623143 4560 scope.go:122] "RemoveContainer" containerID="f6de60c92d5a83f37e533540b8ffc097c3617fa8ef986252c61a62d4c278aa76" Oct 30 04:00:17.637419 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.637390 4560 scope.go:122] "RemoveContainer" containerID="628219daa7e2216b097eb5aa841b55065651bc32911611342718c365f40effc0" Oct 30 04:00:17.650056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.650026 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 04:00:17.656112 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.656088 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e0d09-c76884bc-l8j2r"] Oct 30 04:00:17.760177 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.760150 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:17.767164 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.767132 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888880-462mm"] Oct 30 04:00:17.781012 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.780981 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:17.783773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.783744 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 04:00:17.788597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.788574 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-lv6bh"] Oct 30 04:00:17.879191 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.879095 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\" (UniqueName: \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\") pod \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " Oct 30 04:00:17.879363 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.879320 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\") pod \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\" (UID: \"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8\") " Oct 30 04:00:17.879778 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.879744 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca" pod "8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" (UID: "8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:17.880018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.879996 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:17.881840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.881814 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26" pod "8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" (UID: "8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8"). InnerVolumeSpecName "kube-api-access-5bt26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:17.981086 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:17.981049 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bt26\" (UniqueName: \"kubernetes.io/projected/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8-kube-api-access-5bt26\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.156126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.156023 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" path="/var/lib/kubelet/pods/304dac9a-5763-475a-9107-a12d703cb8f0/volumes" Oct 30 04:00:18.156849 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.156807 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" path="/var/lib/kubelet/pods/64fddf95-b5e1-4d79-a65f-92fb3b6e1295/volumes" Oct 30 04:00:18.157409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.157391 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73ccabf-284e-4670-b12b-0df010cf5aed" path="/var/lib/kubelet/pods/d73ccabf-284e-4670-b12b-0df010cf5aed/volumes" Oct 30 04:00:18.606872 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.606842 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" Oct 30 04:00:18.607359 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.606838 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr" event={"ID":"8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8","Type":"ContainerDied","Data":"20a39c227a4e2d8b17fc3384830f33126993686dfcc3b325a54517caed72bf8e"} Oct 30 04:00:18.607359 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.606984 4560 scope.go:122] "RemoveContainer" containerID="a62e1af04d8ff3da79ac8a72e77cb4a913df702f4cffdb74afb0d92b0d494a42" Oct 30 04:00:18.609994 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.609973 4560 generic.go:356] "Generic (PLEG): container finished" podID="03893764-1fb6-43de-9972-917cc084c474" containerID="f8b9f816228feeee447c3487899f42ce92d54ca2020459719b288daf02ca012c" exitCode=60 Oct 30 04:00:18.610075 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.610041 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerDied","Data":"f8b9f816228feeee447c3487899f42ce92d54ca2020459719b288daf02ca012c"} Oct 30 04:00:18.610452 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.610435 4560 scope.go:122] "RemoveContainer" containerID="f8b9f816228feeee447c3487899f42ce92d54ca2020459719b288daf02ca012c" Oct 30 04:00:18.610823 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:18.610797 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7_odh-ai-gateway-infra(03893764-1fb6-43de-9972-917cc084c474)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" podUID="03893764-1fb6-43de-9972-917cc084c474" Oct 30 04:00:18.617885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.617865 4560 scope.go:122] "RemoveContainer" containerID="48ae7117f3b875ad314093e408d8ed21cbc0cb33a0a26ac78dbe2dd27797ade7" Oct 30 04:00:18.659573 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.659499 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:18.662763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:18.662734 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0393ed-29888880-fbrpr"] Oct 30 04:00:19.711237 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:19.711202 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:19.870325 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:19.870300 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:20.009170 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.009078 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\" (UniqueName: \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\") pod \"03893764-1fb6-43de-9972-917cc084c474\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " Oct 30 04:00:20.009360 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.009168 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\") pod \"03893764-1fb6-43de-9972-917cc084c474\" (UID: \"03893764-1fb6-43de-9972-917cc084c474\") " Oct 30 04:00:20.009535 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.009481 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca" pod "03893764-1fb6-43de-9972-917cc084c474" (UID: "03893764-1fb6-43de-9972-917cc084c474"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:20.009804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.009785 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03893764-1fb6-43de-9972-917cc084c474-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.011451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.011430 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26" pod "03893764-1fb6-43de-9972-917cc084c474" (UID: "03893764-1fb6-43de-9972-917cc084c474"). InnerVolumeSpecName "kube-api-access-2vl26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:20.111487 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.111443 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vl26\" (UniqueName: \"kubernetes.io/projected/03893764-1fb6-43de-9972-917cc084c474-kube-api-access-2vl26\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.157429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.157389 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" path="/var/lib/kubelet/pods/8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8/volumes" Oct 30 04:00:20.627233 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.627202 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" Oct 30 04:00:20.627418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.627236 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7" event={"ID":"03893764-1fb6-43de-9972-917cc084c474","Type":"ContainerDied","Data":"7f25b2ee65821735d823251ad3b754080d01d30fdfa13a1d4a73639be3be3bf0"} Oct 30 04:00:20.627418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.627293 4560 scope.go:122] "RemoveContainer" containerID="f8b9f816228feeee447c3487899f42ce92d54ca2020459719b288daf02ca012c" Oct 30 04:00:20.681634 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.681591 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:20.694002 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:20.693975 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-81afe3-29888880-z6gw7"] Oct 30 04:00:22.043270 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.043220 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" podUID="b9b2c191-18bc-4974-b952-155fd3761ff9" containerName="main" containerID="cri-o://11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793" gracePeriod=45 Oct 30 04:00:22.156745 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.156717 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03893764-1fb6-43de-9972-917cc084c474" path="/var/lib/kubelet/pods/03893764-1fb6-43de-9972-917cc084c474/volumes" Oct 30 04:00:22.344757 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.344728 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 04:00:22.440068 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440037 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440111 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440137 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440232 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440442 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440350 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440442 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440420 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:22.440442 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440438 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\") pod \"b9b2c191-18bc-4974-b952-155fd3761ff9\" (UID: \"b9b2c191-18bc-4974-b952-155fd3761ff9\") " Oct 30 04:00:22.440641 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440601 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:22.440696 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.440667 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:22.441037 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.441017 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.441136 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.441054 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.441136 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.441063 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.442553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.442494 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:22.442646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.442610 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:22.442646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.442618 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm" pod "b9b2c191-18bc-4974-b952-155fd3761ff9" (UID: "b9b2c191-18bc-4974-b952-155fd3761ff9"). InnerVolumeSpecName "kube-api-access-xvcwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:22.542341 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.542308 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b2c191-18bc-4974-b952-155fd3761ff9-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.542341 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.542337 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/b9b2c191-18bc-4974-b952-155fd3761ff9-kube-api-access-xvcwm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.542341 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.542348 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b2c191-18bc-4974-b952-155fd3761ff9-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:22.643664 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.643584 4560 generic.go:356] "Generic (PLEG): container finished" podID="b9b2c191-18bc-4974-b952-155fd3761ff9" containerID="11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793" exitCode=0 Oct 30 04:00:22.643664 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.643619 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" event={"ID":"b9b2c191-18bc-4974-b952-155fd3761ff9","Type":"ContainerDied","Data":"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793"} Oct 30 04:00:22.643664 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.643648 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" Oct 30 04:00:22.643908 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.643680 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b" event={"ID":"b9b2c191-18bc-4974-b952-155fd3761ff9","Type":"ContainerDied","Data":"0da57678695a2cf465a29cac587c89ee5a9bccf76f6cb20ff1096308a0563b13"} Oct 30 04:00:22.643908 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.643706 4560 scope.go:122] "RemoveContainer" containerID="11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793" Oct 30 04:00:22.658188 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.658160 4560 scope.go:122] "RemoveContainer" containerID="11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793" Oct 30 04:00:22.658613 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:22.658583 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793\": container with ID starting with 11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793 not found: ID does not exist" containerID="11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793" Oct 30 04:00:22.658697 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.658626 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793"} err="failed to get container status \"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793\": rpc error: code = NotFound desc = could not find container \"11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793\": container with ID starting with 11794aa1fb5d2848260acee9142fb0ec965400a59e8470a6046c2086bd856793 not found: ID does not exist" Oct 30 04:00:22.699653 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.699624 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 04:00:22.713106 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:22.713071 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4206d9/test-model-4206d9-kserve-866cd87b7c-pxn9b"] Oct 30 04:00:23.147539 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:23.146018 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 04:00:23.147539 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:23.146232 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" containerID="cri-o://1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492" gracePeriod=30 Oct 30 04:00:23.162619 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:23.162397 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 04:00:23.162782 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:23.162639 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" containerID="cri-o://778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5" gracePeriod=30 Oct 30 04:00:24.154650 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:24.154618 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b2c191-18bc-4974-b952-155fd3761ff9" path="/var/lib/kubelet/pods/b9b2c191-18bc-4974-b952-155fd3761ff9/volumes" Oct 30 04:00:28.467097 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.467067 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 04:00:28.483874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.483849 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 04:00:28.533623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.533592 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.533838 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.533813 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.533914 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.533873 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.533969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.533911 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.533969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.533940 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.534089 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.534008 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.534089 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.534062 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.534203 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.534095 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.534203 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.534132 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.534203 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.534194 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536074 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536315 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536313 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536246 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536561 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536617 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.536646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.536651 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:28.537358 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.537327 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:28.537492 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.537379 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.537492 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.537481 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.538213 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538177 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "kube-api-access-vh99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.538659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538636 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.538787 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538728 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.538885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538787 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.538885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538828 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.539056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538884 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\") pod \"790293ab-1b10-49eb-b35f-89d3ea4fac61\" (UID: \"790293ab-1b10-49eb-b35f-89d3ea4fac61\") " Oct 30 04:00:28.539056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.538959 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\") pod \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\" (UID: \"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2\") " Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.539551 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.539737 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540262 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540287 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790293ab-1b10-49eb-b35f-89d3ea4fac61-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540302 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540424 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540442 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540455 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540538 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540556 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh99r\" (UniqueName: \"kubernetes.io/projected/790293ab-1b10-49eb-b35f-89d3ea4fac61-kube-api-access-vh99r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540571 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.540985 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.542017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.541951 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "kube-api-access-5w4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.543172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.542687 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.543172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.542761 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:28.543172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.542827 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.543652 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.543631 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.544937 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.544910 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo" pod "790293ab-1b10-49eb-b35f-89d3ea4fac61" (UID: "790293ab-1b10-49eb-b35f-89d3ea4fac61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:28.546251 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.546231 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy" pod "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" (UID: "fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.642059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642021 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642057 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642070 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642083 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4ld\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-kube-api-access-5w4ld\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642095 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642107 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790293ab-1b10-49eb-b35f-89d3ea4fac61-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642119 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.642271 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.642131 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.689534 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.689494 4560 generic.go:356] "Generic (PLEG): container finished" podID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerID="778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5" exitCode=0 Oct 30 04:00:28.689729 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.689578 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" Oct 30 04:00:28.689729 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.689573 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" event={"ID":"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2","Type":"ContainerDied","Data":"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5"} Oct 30 04:00:28.689729 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.689730 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74" event={"ID":"fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2","Type":"ContainerDied","Data":"61a5e75d5543ca358b7d64539de9229b94240752b529f2428a282dae7e213f40"} Oct 30 04:00:28.689908 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.689764 4560 scope.go:122] "RemoveContainer" containerID="778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5" Oct 30 04:00:28.691094 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.691071 4560 generic.go:356] "Generic (PLEG): container finished" podID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerID="1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492" exitCode=0 Oct 30 04:00:28.691231 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.691104 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" event={"ID":"790293ab-1b10-49eb-b35f-89d3ea4fac61","Type":"ContainerDied","Data":"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492"} Oct 30 04:00:28.691231 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.691127 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" Oct 30 04:00:28.691231 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.691162 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx" event={"ID":"790293ab-1b10-49eb-b35f-89d3ea4fac61","Type":"ContainerDied","Data":"965ccbde7c15bb97bb68e73e120c099158e4bfba5c46f8baa3990d7aec65aae3"} Oct 30 04:00:28.701725 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.701702 4560 scope.go:122] "RemoveContainer" containerID="778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5" Oct 30 04:00:28.702012 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:28.701970 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5\": container with ID starting with 778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5 not found: ID does not exist" containerID="778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5" Oct 30 04:00:28.702012 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.701996 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5"} err="failed to get container status \"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5\": rpc error: code = NotFound desc = could not find container \"778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5\": container with ID starting with 778ff05f1b4d51f050909c981bad053161c44df84120b042a7f40493ef9b07d5 not found: ID does not exist" Oct 30 04:00:28.702150 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.702014 4560 scope.go:122] "RemoveContainer" containerID="1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492" Oct 30 04:00:28.711894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.711876 4560 scope.go:122] "RemoveContainer" containerID="1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492" Oct 30 04:00:28.712161 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:28.712144 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492\": container with ID starting with 1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492 not found: ID does not exist" containerID="1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492" Oct 30 04:00:28.712209 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.712168 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492"} err="failed to get container status \"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492\": rpc error: code = NotFound desc = could not find container \"1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492\": container with ID starting with 1f50579ebd498e366c768de8d2dfbc8a0899ab05cf6e2cf5374530944e566492 not found: ID does not exist" Oct 30 04:00:28.742835 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.742807 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 04:00:28.748943 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.748915 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-cnvwx"] Oct 30 04:00:28.784640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.784612 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 04:00:28.787426 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:28.787404 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tzn74"] Oct 30 04:00:30.153731 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:30.153700 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" path="/var/lib/kubelet/pods/790293ab-1b10-49eb-b35f-89d3ea4fac61/volumes" Oct 30 04:00:30.154239 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:30.154225 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" path="/var/lib/kubelet/pods/fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2/volumes" Oct 30 04:00:41.750870 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:00:41.750824 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:00:56.362894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.362860 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:00:56.363496 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363480 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b9b2c191-18bc-4974-b952-155fd3761ff9" containerName="main" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363496 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b2c191-18bc-4974-b952-155fd3761ff9" containerName="main" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363511 4560 cpu_manager.go:414] "containerMap: removing container" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363529 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363546 4560 cpu_manager.go:414] "containerMap: removing container" podUID="055098bc-6674-4e59-8619-afae8f20d297" containerName="main" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363552 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055098bc-6674-4e59-8619-afae8f20d297" containerName="main" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363561 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.363566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363566 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363581 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d73ccabf-284e-4670-b12b-0df010cf5aed" containerName="maas-api" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363586 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73ccabf-284e-4670-b12b-0df010cf5aed" containerName="maas-api" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363593 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363597 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363614 4560 cpu_manager.go:414] "containerMap: removing container" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363618 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363627 4560 cpu_manager.go:414] "containerMap: removing container" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363632 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363639 4560 cpu_manager.go:414] "containerMap: removing container" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363643 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363649 4560 cpu_manager.go:414] "containerMap: removing container" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363653 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363659 4560 cpu_manager.go:414] "containerMap: removing container" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363664 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363673 4560 cpu_manager.go:414] "containerMap: removing container" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363677 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363686 4560 cpu_manager.go:414] "containerMap: removing container" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363691 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363697 4560 cpu_manager.go:414] "containerMap: removing container" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.363786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363702 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363799 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363809 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73ccabf-284e-4670-b12b-0df010cf5aed" containerName="maas-api" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363817 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fddf95-b5e1-4d79-a65f-92fb3b6e1295" containerName="istio-proxy" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363824 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363831 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363835 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0bfe6a-98d6-4e1c-8ca9-f5d347b05bb2" containerName="istio-proxy" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363841 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363848 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="790293ab-1b10-49eb-b35f-89d3ea4fac61" containerName="istio-proxy" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363853 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="636fe6a3-c210-4ad2-8f64-80a61125d145" containerName="payload-pre-processing" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363858 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="03893764-1fb6-43de-9972-917cc084c474" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363864 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363871 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b2c191-18bc-4974-b952-155fd3761ff9" containerName="main" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363876 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.364366 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.363881 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="055098bc-6674-4e59-8619-afae8f20d297" containerName="main" Oct 30 04:00:56.367451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.367430 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.386108 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.386070 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:00:56.394961 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.394934 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1597f8bc\"" Oct 30 04:00:56.394961 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.394949 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1597f8bc-dockercfg-xw8hj\"" Oct 30 04:00:56.463089 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.463051 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:00:56.464157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464133 4560 cpu_manager.go:414] "containerMap: removing container" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.464242 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464160 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.464242 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464200 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.464242 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464208 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.464427 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464412 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="304dac9a-5763-475a-9107-a12d703cb8f0" containerName="cleanup" Oct 30 04:00:56.464467 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.464437 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc8fb6-bd2d-4e0e-bdb9-8659cf5330d8" containerName="cleanup" Oct 30 04:00:56.468802 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.468780 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.493830 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.493793 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:00:56.521255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.521195 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8cq\" (UniqueName: \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.521586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.521550 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.622885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.622796 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.622885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.622880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.623083 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.622910 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8cq\" (UniqueName: \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.623083 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.622952 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfjq\" (UniqueName: \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.623645 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.623627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.642434 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.642401 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8cq\" (UniqueName: \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\") pod \"payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.681258 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.681227 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:56.724160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.724127 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.724312 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.724212 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfjq\" (UniqueName: \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.724913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.724891 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.753749 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.745036 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfjq\" (UniqueName: \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\") pod \"payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.799018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.798986 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:56.986588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:56.986556 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:00:57.106382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.106355 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:00:57.107203 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:00:57.107181 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5d0562_e79e_4d7a_a88c_16d25c577da0.slice/crio-d02411fdc4a74b8bebd944dc6641326d388dd73696fe5f9719bd3940bdd2e153 WatchSource:0}: Error finding container d02411fdc4a74b8bebd944dc6641326d388dd73696fe5f9719bd3940bdd2e153: Status 404 returned error can't find the container with id d02411fdc4a74b8bebd944dc6641326d388dd73696fe5f9719bd3940bdd2e153 Oct 30 04:00:57.937848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.937807 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" event={"ID":"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2","Type":"ContainerStarted","Data":"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f"} Oct 30 04:00:57.938298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.937873 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" event={"ID":"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2","Type":"ContainerStarted","Data":"77815b7799a0cd6ed5f63bb0e5b27aabd441277f5d831e3a84533d8cf1f2d49d"} Oct 30 04:00:57.938298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.937905 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:00:57.939542 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.939503 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" event={"ID":"fe5d0562-e79e-4d7a-a88c-16d25c577da0","Type":"ContainerStarted","Data":"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2"} Oct 30 04:00:57.939650 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.939557 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" event={"ID":"fe5d0562-e79e-4d7a-a88c-16d25c577da0","Type":"ContainerStarted","Data":"d02411fdc4a74b8bebd944dc6641326d388dd73696fe5f9719bd3940bdd2e153"} Oct 30 04:00:57.939650 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.939577 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:00:57.981350 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:57.981284 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" podStartSLOduration=1.981265025 podStartE2EDuration="1.981265025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:57.976982362 +0000 UTC m=+3485.962102247" watchObservedRunningTime="2026-10-30 04:00:57.981265025 +0000 UTC m=+3485.966384915" Oct 30 04:00:58.021801 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:00:58.021731 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" podStartSLOduration=2.021716603 podStartE2EDuration="2.021716603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:58.01632328 +0000 UTC m=+3486.001443169" watchObservedRunningTime="2026-10-30 04:00:58.021716603 +0000 UTC m=+3486.006836493" Oct 30 04:01:08.949143 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:08.949112 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:01:08.949591 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:08.949183 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:01:11.753248 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:01:11.753199 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:01:24.071785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.071747 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 04:01:24.073020 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.072097 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" podUID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" containerName="maas-api" containerID="cri-o://1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971" gracePeriod=30 Oct 30 04:01:24.394451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.394423 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 04:01:24.438545 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.438482 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\") pod \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " Oct 30 04:01:24.438915 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.438644 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\" (UniqueName: \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\") pod \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " Oct 30 04:01:24.438915 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.438694 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\") pod \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " Oct 30 04:01:24.438915 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.438774 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\") pod \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\" (UID: \"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71\") " Oct 30 04:01:24.439532 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.439490 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca" pod "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" (UID: "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:01:24.442497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.442466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs" pod "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" (UID: "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:01:24.442763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.442734 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls" pod "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" (UID: "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:01:24.454718 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.452086 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj" pod "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" (UID: "ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71"). InnerVolumeSpecName "kube-api-access-fsqbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:24.541098 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.541061 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:01:24.541098 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.541095 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqbj\" (UniqueName: \"kubernetes.io/projected/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-kube-api-access-fsqbj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:01:24.541305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.541109 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:01:24.541305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:24.541122 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:01:25.130339 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.130304 4560 generic.go:356] "Generic (PLEG): container finished" podID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" containerID="1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971" exitCode=0 Oct 30 04:01:25.130791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.130358 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" Oct 30 04:01:25.130791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.130394 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" event={"ID":"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71","Type":"ContainerDied","Data":"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971"} Oct 30 04:01:25.130791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.130450 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2" event={"ID":"ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71","Type":"ContainerDied","Data":"27e9d34aa27c9a44832858787ecc4d57c336420df0bc98a84d463ff3debe9bba"} Oct 30 04:01:25.130791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.130472 4560 scope.go:122] "RemoveContainer" containerID="1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971" Oct 30 04:01:25.141162 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.141143 4560 scope.go:122] "RemoveContainer" containerID="1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971" Oct 30 04:01:25.141410 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:01:25.141391 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971\": container with ID starting with 1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971 not found: ID does not exist" containerID="1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971" Oct 30 04:01:25.141473 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.141418 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971"} err="failed to get container status \"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971\": rpc error: code = NotFound desc = could not find container \"1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971\": container with ID starting with 1a8eca37a71e7c0d66ff3cdcfacb7fd65e9b07923dcef33bf4cd840405359971 not found: ID does not exist" Oct 30 04:01:25.184553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.184497 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 04:01:25.195116 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:25.195079 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4206d9-7cb4bdb5fc-xm9f2"] Oct 30 04:01:26.155195 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:26.155158 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" path="/var/lib/kubelet/pods/ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71/volumes" Oct 30 04:01:34.388718 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.388676 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:01:34.389363 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.389346 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" containerName="maas-api" Oct 30 04:01:34.389363 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.389363 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" containerName="maas-api" Oct 30 04:01:34.389495 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.389486 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbde9d0-7d76-43d4-abc1-bf2bc2d30e71" containerName="maas-api" Oct 30 04:01:34.393300 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.393281 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.422976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.422881 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1597f8bc-serving-cert\"" Oct 30 04:01:34.428457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.428424 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:01:34.467562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.467506 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.467730 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.467601 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.467730 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.467670 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.467811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.467763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd78q\" (UniqueName: \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.569056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.569019 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.569056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.569054 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.569268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.569115 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.569268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.569196 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd78q\" (UniqueName: \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.569917 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.569891 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.571741 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.571711 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.571834 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.571720 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.589194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.589162 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd78q\" (UniqueName: \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\") pod \"maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:34.707137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:34.707104 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:35.016033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:35.015858 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:01:35.025972 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:35.025941 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:01:35.200965 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:35.200923 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" event={"ID":"cf3d0b20-29c3-494e-81ab-dfdc114a599f","Type":"ContainerStarted","Data":"8363d8fd228e864097002e1ca0489fd294b9e43f10bc961d940fc2762c6658f7"} Oct 30 04:01:36.208935 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:36.208899 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" event={"ID":"cf3d0b20-29c3-494e-81ab-dfdc114a599f","Type":"ContainerStarted","Data":"f3f188b5587247521b28832d5dafc2d0c8c1f7a593329f88cf70d217fa6d2562"} Oct 30 04:01:36.209427 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:36.208989 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:36.250552 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:36.250457 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" podStartSLOduration=1.877921324 podStartE2EDuration="2.250439553s" totalImagesPullingTime="372.518229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:36.249431517 +0000 UTC m=+3524.234551405" watchObservedRunningTime="2026-10-30 04:01:36.250439553 +0000 UTC m=+3524.235559443" Oct 30 04:01:38.290224 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:38.290188 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 04:01:41.750655 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:01:41.750615 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:01:42.221384 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:42.221353 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:01:43.290623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:43.290563 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" containerID="cri-o://cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f" gracePeriod=25 Oct 30 04:01:48.810773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.810730 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:01:48.815530 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.815497 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.835033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.835002 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:01:48.843285 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.843259 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"openshift-service-ca.crt\"" Oct 30 04:01:48.843426 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.843361 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"kube-root-ca.crt\"" Oct 30 04:01:48.843426 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.843368 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"sub-test-model-da4553-kserve-self-signed-certs\"" Oct 30 04:01:48.843637 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.843617 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-da4553\"/\"default-dockercfg-mwqzk\"" Oct 30 04:01:48.960000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.959959 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.960183 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.960019 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.960183 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.960147 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7974\" (UniqueName: \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.960275 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.960199 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.960344 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.960317 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:48.960394 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:48.960369 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061575 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7974\" (UniqueName: \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061638 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061695 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061727 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061901 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.061979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.061951 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.062383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.062353 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.062712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.062684 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.062848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.062822 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.065606 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.065579 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.072171 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.072117 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.082017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.081985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7974\" (UniqueName: \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\") pod \"sub-test-model-da4553-kserve-5f87b865d9-6nj75\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.131030 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.130993 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:49.440732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:49.440694 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:01:49.441094 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:01:49.441072 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac924b5e_268a_45b6_bf71_3d7b0dde8491.slice/crio-30a34a68f25a4cbdb5a698341a0216e45946bb52cca990ed26ec46d52cc55842 WatchSource:0}: Error finding container 30a34a68f25a4cbdb5a698341a0216e45946bb52cca990ed26ec46d52cc55842: Status 404 returned error can't find the container with id 30a34a68f25a4cbdb5a698341a0216e45946bb52cca990ed26ec46d52cc55842 Oct 30 04:01:50.079185 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:50.079144 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:01:50.300779 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:50.300738 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" event={"ID":"ac924b5e-268a-45b6-bf71-3d7b0dde8491","Type":"ContainerStarted","Data":"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a"} Oct 30 04:01:50.300995 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:50.300791 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" event={"ID":"ac924b5e-268a-45b6-bf71-3d7b0dde8491","Type":"ContainerStarted","Data":"30a34a68f25a4cbdb5a698341a0216e45946bb52cca990ed26ec46d52cc55842"} Oct 30 04:01:50.344106 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:50.344007 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" podStartSLOduration=2.343992474 podStartE2EDuration="2.343992474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:50.3431588 +0000 UTC m=+3538.328278687" watchObservedRunningTime="2026-10-30 04:01:50.343992474 +0000 UTC m=+3538.329112361" Oct 30 04:01:51.131459 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:51.131426 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:59.131394 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:59.131358 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:59.145581 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:59.145547 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:01:59.375965 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:01:59.375929 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:02:00.054289 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.054248 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 04:02:00.054749 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.054554 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" podUID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" containerName="maas-api" containerID="cri-o://73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124" gracePeriod=30 Oct 30 04:02:00.079477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.079111 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:02:00.574683 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.574652 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 04:02:00.624288 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.624214 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\" (UniqueName: \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\") pod \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " Oct 30 04:02:00.624439 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.624341 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\") pod \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " Oct 30 04:02:00.624439 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.624376 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\") pod \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " Oct 30 04:02:00.624580 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.624458 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\") pod \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\" (UID: \"ad3a2a16-0930-4f67-ae0e-7da4887abbd4\") " Oct 30 04:02:00.624870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.624828 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca" pod "ad3a2a16-0930-4f67-ae0e-7da4887abbd4" (UID: "ad3a2a16-0930-4f67-ae0e-7da4887abbd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:00.625093 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.625035 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:00.626632 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.626596 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw" pod "ad3a2a16-0930-4f67-ae0e-7da4887abbd4" (UID: "ad3a2a16-0930-4f67-ae0e-7da4887abbd4"). InnerVolumeSpecName "kube-api-access-pg8nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:00.626759 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.626647 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls" pod "ad3a2a16-0930-4f67-ae0e-7da4887abbd4" (UID: "ad3a2a16-0930-4f67-ae0e-7da4887abbd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:00.627705 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.627678 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs" pod "ad3a2a16-0930-4f67-ae0e-7da4887abbd4" (UID: "ad3a2a16-0930-4f67-ae0e-7da4887abbd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:00.726030 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.725988 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:00.726030 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.726026 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8nw\" (UniqueName: \"kubernetes.io/projected/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-kube-api-access-pg8nw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:00.726030 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.726037 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad3a2a16-0930-4f67-ae0e-7da4887abbd4-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:00.806643 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.806608 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:02:00.807661 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.807629 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" containerName="maas-api" Oct 30 04:02:00.807661 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.807658 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" containerName="maas-api" Oct 30 04:02:00.807896 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.807881 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" containerName="maas-api" Oct 30 04:02:00.811500 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.811478 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827081 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgnq\" (UniqueName: \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827220 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827257 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827289 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.827670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.827445 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.828641 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.828614 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"kube-root-ca.crt\"" Oct 30 04:02:00.828796 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.828707 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"sub-test-model-1a1b45-kserve-self-signed-certs\"" Oct 30 04:02:00.828796 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.828751 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"openshift-service-ca.crt\"" Oct 30 04:02:00.828934 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.828810 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-1a1b45\"/\"default-dockercfg-z9vxb\"" Oct 30 04:02:00.829223 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.829198 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:02:00.929196 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929085 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929196 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929184 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgnq\" (UniqueName: \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929465 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929214 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929465 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929244 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929465 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929270 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929465 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929407 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929724 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929539 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929724 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929643 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.929916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.929867 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.932150 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.932121 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.932390 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.932370 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:00.948794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:00.948759 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgnq\" (UniqueName: \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\") pod \"sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:01.129875 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.129843 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:01.375824 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.375790 4560 generic.go:356] "Generic (PLEG): container finished" podID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" containerID="73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124" exitCode=0 Oct 30 04:02:01.376013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.375840 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" Oct 30 04:02:01.376013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.375863 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" event={"ID":"ad3a2a16-0930-4f67-ae0e-7da4887abbd4","Type":"ContainerDied","Data":"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124"} Oct 30 04:02:01.376013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.375927 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774" event={"ID":"ad3a2a16-0930-4f67-ae0e-7da4887abbd4","Type":"ContainerDied","Data":"e80d5ea3cc3bb539b56662e5dfae3bccc58163581737e7b476d84bc67022e234"} Oct 30 04:02:01.376013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.375959 4560 scope.go:122] "RemoveContainer" containerID="73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124" Oct 30 04:02:01.386532 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.386496 4560 scope.go:122] "RemoveContainer" containerID="73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124" Oct 30 04:02:01.386803 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:02:01.386785 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124\": container with ID starting with 73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124 not found: ID does not exist" containerID="73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124" Oct 30 04:02:01.386876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.386812 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124"} err="failed to get container status \"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124\": rpc error: code = NotFound desc = could not find container \"73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124\": container with ID starting with 73d0b436f76abdbd69d1990be490591478c94eca83cf9f4375ea7ef2b579f124 not found: ID does not exist" Oct 30 04:02:01.428401 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.428365 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 04:02:01.441092 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.441067 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0393ed-5888cbd9b9-4f774"] Oct 30 04:02:01.441504 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:02:01.441478 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30801a51_1c6b_40ee_9540_4c94d37769de.slice/crio-7b3becdf2f62571f37e8b839e3ef5259ffdf37c7c559eb4389c4cd4824218e2c WatchSource:0}: Error finding container 7b3becdf2f62571f37e8b839e3ef5259ffdf37c7c559eb4389c4cd4824218e2c: Status 404 returned error can't find the container with id 7b3becdf2f62571f37e8b839e3ef5259ffdf37c7c559eb4389c4cd4824218e2c Oct 30 04:02:01.444675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:01.444651 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:02:02.168232 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:02.168197 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3a2a16-0930-4f67-ae0e-7da4887abbd4" path="/var/lib/kubelet/pods/ad3a2a16-0930-4f67-ae0e-7da4887abbd4/volumes" Oct 30 04:02:02.386104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:02.386065 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" event={"ID":"30801a51-1c6b-40ee-9540-4c94d37769de","Type":"ContainerStarted","Data":"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49"} Oct 30 04:02:02.386315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:02.386117 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" event={"ID":"30801a51-1c6b-40ee-9540-4c94d37769de","Type":"ContainerStarted","Data":"7b3becdf2f62571f37e8b839e3ef5259ffdf37c7c559eb4389c4cd4824218e2c"} Oct 30 04:02:02.429002 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:02.428900 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" podStartSLOduration=2.428886034 podStartE2EDuration="2.428886034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:02.427368811 +0000 UTC m=+3550.412488712" watchObservedRunningTime="2026-10-30 04:02:02.428886034 +0000 UTC m=+3550.414005953" Oct 30 04:02:03.130549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.130486 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:03.566874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.566846 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 04:02:03.668595 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.668480 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\") pod \"4bce743b-8233-4ac1-91a8-8eb11bc75238\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " Oct 30 04:02:03.668762 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.668728 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\" (UniqueName: \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\") pod \"4bce743b-8233-4ac1-91a8-8eb11bc75238\" (UID: \"4bce743b-8233-4ac1-91a8-8eb11bc75238\") " Oct 30 04:02:03.668972 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.668943 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume" pod "4bce743b-8233-4ac1-91a8-8eb11bc75238" (UID: "4bce743b-8233-4ac1-91a8-8eb11bc75238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:03.669161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.669143 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce743b-8233-4ac1-91a8-8eb11bc75238-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:03.670963 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.670934 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2" pod "4bce743b-8233-4ac1-91a8-8eb11bc75238" (UID: "4bce743b-8233-4ac1-91a8-8eb11bc75238"). InnerVolumeSpecName "kube-api-access-zbgn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:03.770223 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:03.770182 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgn2\" (UniqueName: \"kubernetes.io/projected/4bce743b-8233-4ac1-91a8-8eb11bc75238-kube-api-access-zbgn2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:02:04.401670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.401636 4560 generic.go:356] "Generic (PLEG): container finished" podID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerID="cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f" exitCode=0 Oct 30 04:02:04.401858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.401687 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" Oct 30 04:02:04.401858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.401707 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" event={"ID":"4bce743b-8233-4ac1-91a8-8eb11bc75238","Type":"ContainerDied","Data":"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f"} Oct 30 04:02:04.401858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.401769 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h" event={"ID":"4bce743b-8233-4ac1-91a8-8eb11bc75238","Type":"ContainerDied","Data":"bb6c8ad363d7606eae251c457f21faf63cb04a34063f8fb7649fb4f8a0d91ff5"} Oct 30 04:02:04.401858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.401801 4560 scope.go:122] "RemoveContainer" containerID="cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f" Oct 30 04:02:04.412908 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.412685 4560 scope.go:122] "RemoveContainer" containerID="cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f" Oct 30 04:02:04.413016 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:02:04.412994 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f\": container with ID starting with cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f not found: ID does not exist" containerID="cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f" Oct 30 04:02:04.413061 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.413028 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f"} err="failed to get container status \"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f\": rpc error: code = NotFound desc = could not find container \"cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f\": container with ID starting with cc8f4e530fd9303fa72a4f37385573d9a1470c4541c31d30d40a882a5d361c8f not found: ID does not exist" Oct 30 04:02:04.455071 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.455038 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 04:02:04.467142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:04.467116 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0393ed-7c7846fbcb-nq52h"] Oct 30 04:02:06.154825 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:06.154790 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" path="/var/lib/kubelet/pods/4bce743b-8233-4ac1-91a8-8eb11bc75238/volumes" Oct 30 04:02:11.130069 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:11.130035 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:11.143278 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:11.143249 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:11.467534 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:02:11.467486 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:02:11.751938 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:02:11.751856 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:02:41.750612 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:02:41.750576 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:03:11.750774 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:03:11.750732 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:03:41.750494 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:03:41.750415 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:03:59.791088 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:03:59.791047 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:03:59.824971 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:03:59.824930 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:04:04.791373 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:04.791331 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" podUID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" containerName="payload-processing" containerID="cri-o://1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2" gracePeriod=25 Oct 30 04:04:04.826044 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:04.825997 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" podUID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" containerName="payload-pre-processing" containerID="cri-o://7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f" gracePeriod=25 Oct 30 04:04:05.119504 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.119478 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:04:05.156350 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.156325 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:04:05.234670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.234625 4560 generic.go:356] "Generic (PLEG): container finished" podID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" containerID="1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2" exitCode=0 Oct 30 04:04:05.234870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.234679 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" Oct 30 04:04:05.234870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.234716 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" event={"ID":"fe5d0562-e79e-4d7a-a88c-16d25c577da0","Type":"ContainerDied","Data":"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2"} Oct 30 04:04:05.234870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.234779 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k" event={"ID":"fe5d0562-e79e-4d7a-a88c-16d25c577da0","Type":"ContainerDied","Data":"d02411fdc4a74b8bebd944dc6641326d388dd73696fe5f9719bd3940bdd2e153"} Oct 30 04:04:05.234870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.234810 4560 scope.go:122] "RemoveContainer" containerID="1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2" Oct 30 04:04:05.237078 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.237055 4560 generic.go:356] "Generic (PLEG): container finished" podID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" containerID="7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f" exitCode=0 Oct 30 04:04:05.237422 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.237344 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" event={"ID":"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2","Type":"ContainerDied","Data":"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f"} Oct 30 04:04:05.237422 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.237400 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" event={"ID":"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2","Type":"ContainerDied","Data":"77815b7799a0cd6ed5f63bb0e5b27aabd441277f5d831e3a84533d8cf1f2d49d"} Oct 30 04:04:05.237813 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.237675 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd" Oct 30 04:04:05.257665 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.257637 4560 scope.go:122] "RemoveContainer" containerID="1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2" Oct 30 04:04:05.258027 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:05.258004 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2\": container with ID starting with 1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2 not found: ID does not exist" containerID="1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2" Oct 30 04:04:05.258122 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.258038 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2"} err="failed to get container status \"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2\": rpc error: code = NotFound desc = could not find container \"1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2\": container with ID starting with 1b5fd0879b5be711bc844e587c928f5f78236af47323be2ab69f098c87955ad2 not found: ID does not exist" Oct 30 04:04:05.258122 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.258061 4560 scope.go:122] "RemoveContainer" containerID="7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f" Oct 30 04:04:05.268676 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.268631 4560 scope.go:122] "RemoveContainer" containerID="7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f" Oct 30 04:04:05.268980 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:05.268938 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f\": container with ID starting with 7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f not found: ID does not exist" containerID="7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f" Oct 30 04:04:05.269033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.268988 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f"} err="failed to get container status \"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f\": rpc error: code = NotFound desc = could not find container \"7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f\": container with ID starting with 7d821e9fd985feb21d1ad979eb56872043e046f4c039edf7d2facf2eff68805f not found: ID does not exist" Oct 30 04:04:05.283418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.283395 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\" (UniqueName: \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\") pod \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " Oct 30 04:04:05.283570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.283436 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\" (UniqueName: \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\") pod \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " Oct 30 04:04:05.283689 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.283673 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\") pod \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\" (UID: \"fe5d0562-e79e-4d7a-a88c-16d25c577da0\") " Oct 30 04:04:05.283776 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.283763 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\") pod \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\" (UID: \"89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2\") " Oct 30 04:04:05.284119 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.284093 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume" pod "fe5d0562-e79e-4d7a-a88c-16d25c577da0" (UID: "fe5d0562-e79e-4d7a-a88c-16d25c577da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:05.284208 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.284129 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume" pod "89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" (UID: "89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:05.284573 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.284551 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5d0562-e79e-4d7a-a88c-16d25c577da0-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:05.284677 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.284578 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:05.285572 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.285536 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq" pod "fe5d0562-e79e-4d7a-a88c-16d25c577da0" (UID: "fe5d0562-e79e-4d7a-a88c-16d25c577da0"). InnerVolumeSpecName "kube-api-access-flfjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:05.285668 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.285641 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq" pod "89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" (UID: "89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2"). InnerVolumeSpecName "kube-api-access-cq8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:05.386530 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.386430 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfjq\" (UniqueName: \"kubernetes.io/projected/fe5d0562-e79e-4d7a-a88c-16d25c577da0-kube-api-access-flfjq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:05.386530 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.386463 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq8cq\" (UniqueName: \"kubernetes.io/projected/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2-kube-api-access-cq8cq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:05.583996 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.583964 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:04:05.586741 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.586719 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1597f8bc-76db4dc74-r4q2k"] Oct 30 04:04:05.620604 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.620575 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:04:05.623652 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:05.623631 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1597f8bc-6c6ccb9475-kgbjd"] Oct 30 04:04:06.154283 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:06.154204 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" path="/var/lib/kubelet/pods/89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2/volumes" Oct 30 04:04:06.154780 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:06.154764 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" path="/var/lib/kubelet/pods/fe5d0562-e79e-4d7a-a88c-16d25c577da0/volumes" Oct 30 04:04:07.971458 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:07.971415 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:04:07.971942 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:07.971713 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" podUID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" containerName="maas-api" containerID="cri-o://f3f188b5587247521b28832d5dafc2d0c8c1f7a593329f88cf70d217fa6d2562" gracePeriod=30 Oct 30 04:04:08.265880 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.265784 4560 generic.go:356] "Generic (PLEG): container finished" podID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" containerID="f3f188b5587247521b28832d5dafc2d0c8c1f7a593329f88cf70d217fa6d2562" exitCode=0 Oct 30 04:04:08.265880 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.265833 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" event={"ID":"cf3d0b20-29c3-494e-81ab-dfdc114a599f","Type":"ContainerDied","Data":"f3f188b5587247521b28832d5dafc2d0c8c1f7a593329f88cf70d217fa6d2562"} Oct 30 04:04:08.300228 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.300200 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:04:08.459400 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.459362 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\") pod \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " Oct 30 04:04:08.459599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.459445 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\" (UniqueName: \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\") pod \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " Oct 30 04:04:08.459599 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.459567 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\") pod \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " Oct 30 04:04:08.459740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.459597 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\") pod \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\" (UID: \"cf3d0b20-29c3-494e-81ab-dfdc114a599f\") " Oct 30 04:04:08.459801 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.459772 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca" pod "cf3d0b20-29c3-494e-81ab-dfdc114a599f" (UID: "cf3d0b20-29c3-494e-81ab-dfdc114a599f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:08.460161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.460133 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d0b20-29c3-494e-81ab-dfdc114a599f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:08.462067 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.462025 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs" pod "cf3d0b20-29c3-494e-81ab-dfdc114a599f" (UID: "cf3d0b20-29c3-494e-81ab-dfdc114a599f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:08.462286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.462251 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q" pod "cf3d0b20-29c3-494e-81ab-dfdc114a599f" (UID: "cf3d0b20-29c3-494e-81ab-dfdc114a599f"). InnerVolumeSpecName "kube-api-access-fd78q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:08.464959 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.464924 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls" pod "cf3d0b20-29c3-494e-81ab-dfdc114a599f" (UID: "cf3d0b20-29c3-494e-81ab-dfdc114a599f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:08.561377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.561297 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd78q\" (UniqueName: \"kubernetes.io/projected/cf3d0b20-29c3-494e-81ab-dfdc114a599f-kube-api-access-fd78q\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:08.561377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.561331 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:08.561377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:08.561340 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d0b20-29c3-494e-81ab-dfdc114a599f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:09.272956 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:09.272921 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" event={"ID":"cf3d0b20-29c3-494e-81ab-dfdc114a599f","Type":"ContainerDied","Data":"8363d8fd228e864097002e1ca0489fd294b9e43f10bc961d940fc2762c6658f7"} Oct 30 04:04:09.272956 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:09.272951 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq" Oct 30 04:04:09.273481 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:09.272990 4560 scope.go:122] "RemoveContainer" containerID="f3f188b5587247521b28832d5dafc2d0c8c1f7a593329f88cf70d217fa6d2562" Oct 30 04:04:09.325526 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:09.325485 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:04:09.339095 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:09.339053 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1597f8bc-7b8769b7cd-n57jq"] Oct 30 04:04:10.153537 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:10.153479 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" path="/var/lib/kubelet/pods/cf3d0b20-29c3-494e-81ab-dfdc114a599f/volumes" Oct 30 04:04:11.751176 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:11.751141 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:04:38.106841 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.106802 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-d95056"] Oct 30 04:04:38.107463 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107446 4560 cpu_manager.go:414] "containerMap: removing container" podUID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" containerName="maas-api" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107463 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" containerName="maas-api" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107489 4560 cpu_manager.go:414] "containerMap: removing container" podUID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" containerName="payload-processing" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107494 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" containerName="payload-processing" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107506 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107511 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107537 4560 cpu_manager.go:414] "containerMap: removing container" podUID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" containerName="payload-pre-processing" Oct 30 04:04:38.107543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107543 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" containerName="payload-pre-processing" Oct 30 04:04:38.107785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107655 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3d0b20-29c3-494e-81ab-dfdc114a599f" containerName="maas-api" Oct 30 04:04:38.107785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107664 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5d0562-e79e-4d7a-a88c-16d25c577da0" containerName="payload-processing" Oct 30 04:04:38.107785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107670 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bce743b-8233-4ac1-91a8-8eb11bc75238" containerName="payload-processing" Oct 30 04:04:38.107785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.107677 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="89aa5c80-2c21-4db4-a3b5-4c4bc122e8d2" containerName="payload-pre-processing" Oct 30 04:04:38.111294 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.111275 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:38.126194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.126161 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-d95056"] Oct 30 04:04:38.153772 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.153731 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 04:04:38.155193 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.154426 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kgh\" (UniqueName: \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\") pod \"test-curl-7573-d95056\" (UID: \"b66c4e15-a802-48b5-a9b0-f882719ba689\") " pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:38.255563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.255497 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kgh\" (UniqueName: \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\") pod \"test-curl-7573-d95056\" (UID: \"b66c4e15-a802-48b5-a9b0-f882719ba689\") " pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:38.275295 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.275265 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kgh\" (UniqueName: \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\") pod \"test-curl-7573-d95056\" (UID: \"b66c4e15-a802-48b5-a9b0-f882719ba689\") " pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:38.425139 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.425053 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:38.727228 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:38.727189 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-d95056"] Oct 30 04:04:38.727799 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:04:38.727765 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb66c4e15_a802_48b5_a9b0_f882719ba689.slice/crio-8671f8f4cea3e36073d36edcaf0bc01cb6e16b6cc17efc164ebf9400f8c12472 WatchSource:0}: Error finding container 8671f8f4cea3e36073d36edcaf0bc01cb6e16b6cc17efc164ebf9400f8c12472: Status 404 returned error can't find the container with id 8671f8f4cea3e36073d36edcaf0bc01cb6e16b6cc17efc164ebf9400f8c12472 Oct 30 04:04:39.471493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:39.471443 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-d95056" event={"ID":"b66c4e15-a802-48b5-a9b0-f882719ba689","Type":"ContainerStarted","Data":"3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6"} Oct 30 04:04:39.471936 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:39.471502 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-d95056" event={"ID":"b66c4e15-a802-48b5-a9b0-f882719ba689","Type":"ContainerStarted","Data":"8671f8f4cea3e36073d36edcaf0bc01cb6e16b6cc17efc164ebf9400f8c12472"} Oct 30 04:04:39.508779 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:39.508731 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7573-d95056" podStartSLOduration=1.508713314 podStartE2EDuration="1.508713314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:39.508271435 +0000 UTC m=+3707.493391322" watchObservedRunningTime="2026-10-30 04:04:39.508713314 +0000 UTC m=+3707.493833204" Oct 30 04:04:40.341588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.341510 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-d95056"] Oct 30 04:04:40.345788 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.345755 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-d95056"] Oct 30 04:04:40.490387 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.490351 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-944894"] Oct 30 04:04:40.491035 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.491018 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" containerName="test-curl-7573-d95056" Oct 30 04:04:40.491093 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.491037 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" containerName="test-curl-7573-d95056" Oct 30 04:04:40.491141 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.491132 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" containerName="test-curl-7573-d95056" Oct 30 04:04:40.494614 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.494597 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:40.510724 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.510701 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-944894"] Oct 30 04:04:40.584060 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.584030 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fjl\" (UniqueName: \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\") pod \"test-curl-7573-944894\" (UID: \"40ffac25-49a6-4f4d-897a-cd5933893c13\") " pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:40.685945 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.685853 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fjl\" (UniqueName: \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\") pod \"test-curl-7573-944894\" (UID: \"40ffac25-49a6-4f4d-897a-cd5933893c13\") " pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:40.707364 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.707334 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fjl\" (UniqueName: \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\") pod \"test-curl-7573-944894\" (UID: \"40ffac25-49a6-4f4d-897a-cd5933893c13\") " pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:40.807256 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:40.807219 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:41.107887 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:41.107852 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-944894"] Oct 30 04:04:41.108471 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:04:41.108438 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ffac25_49a6_4f4d_897a_cd5933893c13.slice/crio-67a491cc9583c03828c488eda4e06816b3c1930d1371db875765b0ccf9f3628f WatchSource:0}: Error finding container 67a491cc9583c03828c488eda4e06816b3c1930d1371db875765b0ccf9f3628f: Status 404 returned error can't find the container with id 67a491cc9583c03828c488eda4e06816b3c1930d1371db875765b0ccf9f3628f Oct 30 04:04:41.485810 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:41.485771 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-944894" event={"ID":"40ffac25-49a6-4f4d-897a-cd5933893c13","Type":"ContainerStarted","Data":"b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2"} Oct 30 04:04:41.486003 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:41.485822 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-944894" event={"ID":"40ffac25-49a6-4f4d-897a-cd5933893c13","Type":"ContainerStarted","Data":"67a491cc9583c03828c488eda4e06816b3c1930d1371db875765b0ccf9f3628f"} Oct 30 04:04:41.486003 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:41.485978 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7573-d95056" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" containerName="test-curl-7573-d95056" containerID="cri-o://3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6" gracePeriod=2 Oct 30 04:04:41.525375 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:41.525325 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7573-944894" podStartSLOduration=1.525311154 podStartE2EDuration="1.525311154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:41.522258259 +0000 UTC m=+3709.507378145" watchObservedRunningTime="2026-10-30 04:04:41.525311154 +0000 UTC m=+3709.510431042" Oct 30 04:04:41.536061 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:41.536032 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-d95056\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:41.750638 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:41.750550 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:04:42.164507 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:42.164422 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-d95056\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:42.343540 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:42.343484 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-944894"] Oct 30 04:04:42.343751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:42.343730 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-944894"] Oct 30 04:04:43.499003 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:43.498967 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7573-944894" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" containerName="test-curl-7573-944894" containerID="cri-o://b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2" gracePeriod=2 Oct 30 04:04:43.513318 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:43.513282 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-944894\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:43.704720 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:43.704691 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:43.718793 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:43.718754 4560 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-944894\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:43.831573 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:43.831472 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\" (UniqueName: \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\") pod \"b66c4e15-a802-48b5-a9b0-f882719ba689\" (UID: \"b66c4e15-a802-48b5-a9b0-f882719ba689\") " Oct 30 04:04:43.833549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:43.833506 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh" pod "b66c4e15-a802-48b5-a9b0-f882719ba689" (UID: "b66c4e15-a802-48b5-a9b0-f882719ba689"). InnerVolumeSpecName "kube-api-access-65kgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:43.932831 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:43.932796 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kgh\" (UniqueName: \"kubernetes.io/projected/b66c4e15-a802-48b5-a9b0-f882719ba689-kube-api-access-65kgh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:44.152876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.152802 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66c4e15-a802-48b5-a9b0-f882719ba689" path="/var/lib/kubelet/pods/b66c4e15-a802-48b5-a9b0-f882719ba689/volumes" Oct 30 04:04:44.506684 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.506648 4560 generic.go:356] "Generic (PLEG): container finished" podID="b66c4e15-a802-48b5-a9b0-f882719ba689" containerID="3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6" exitCode=137 Oct 30 04:04:44.507090 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.506693 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d95056" Oct 30 04:04:44.507090 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.506743 4560 scope.go:122] "RemoveContainer" containerID="3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6" Oct 30 04:04:44.516051 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.516027 4560 scope.go:122] "RemoveContainer" containerID="3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6" Oct 30 04:04:44.516321 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:44.516304 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6\": container with ID starting with 3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6 not found: ID does not exist" containerID="3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6" Oct 30 04:04:44.516380 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:44.516328 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6"} err="failed to get container status \"3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6\": rpc error: code = NotFound desc = could not find container \"3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6\": container with ID starting with 3a09eb691828f648c0a1a00b93d622b5d53133c8eca07666926270c3442f7ae6 not found: ID does not exist" Oct 30 04:04:45.713980 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:45.713949 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:45.856484 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:45.856384 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\" (UniqueName: \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\") pod \"40ffac25-49a6-4f4d-897a-cd5933893c13\" (UID: \"40ffac25-49a6-4f4d-897a-cd5933893c13\") " Oct 30 04:04:45.858839 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:45.858812 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl" pod "40ffac25-49a6-4f4d-897a-cd5933893c13" (UID: "40ffac25-49a6-4f4d-897a-cd5933893c13"). InnerVolumeSpecName "kube-api-access-97fjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:45.957761 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:45.957715 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97fjl\" (UniqueName: \"kubernetes.io/projected/40ffac25-49a6-4f4d-897a-cd5933893c13-kube-api-access-97fjl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:04:46.156942 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.156860 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" path="/var/lib/kubelet/pods/40ffac25-49a6-4f4d-897a-cd5933893c13/volumes" Oct 30 04:04:46.522950 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.522919 4560 generic.go:356] "Generic (PLEG): container finished" podID="40ffac25-49a6-4f4d-897a-cd5933893c13" containerID="b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2" exitCode=137 Oct 30 04:04:46.523160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.522959 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-944894" Oct 30 04:04:46.523160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.523016 4560 scope.go:122] "RemoveContainer" containerID="b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2" Oct 30 04:04:46.532188 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.532172 4560 scope.go:122] "RemoveContainer" containerID="b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2" Oct 30 04:04:46.532435 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:04:46.532415 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2\": container with ID starting with b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2 not found: ID does not exist" containerID="b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2" Oct 30 04:04:46.532489 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:46.532448 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2"} err="failed to get container status \"b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2\": rpc error: code = NotFound desc = could not find container \"b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2\": container with ID starting with b5ba41266e0f96eeb55c0ad52f01dc9efd734131bfb018ee94ee96026c770ff2 not found: ID does not exist" Oct 30 04:04:54.097084 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.097048 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:04:54.097743 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.097727 4560 cpu_manager.go:414] "containerMap: removing container" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" containerName="test-curl-7573-944894" Oct 30 04:04:54.097797 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.097743 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" containerName="test-curl-7573-944894" Oct 30 04:04:54.097861 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.097852 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ffac25-49a6-4f4d-897a-cd5933893c13" containerName="test-curl-7573-944894" Oct 30 04:04:54.103311 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.103289 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.125722 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.125688 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:04:54.127578 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.127500 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-30e3dd-serving-cert\"" Oct 30 04:04:54.172031 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.171981 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.172235 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.172103 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.172235 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.172218 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvbw\" (UniqueName: \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.172376 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.172249 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.274056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.274016 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.274223 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.274106 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.274223 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.274209 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvbw\" (UniqueName: \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.274356 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.274247 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.274941 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.274913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.276754 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.276732 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.277121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.277100 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.295177 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.295146 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvbw\" (UniqueName: \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\") pod \"maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.423786 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.423700 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:54.745156 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:04:54.745122 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cff3e2_50cd_4110_bf7a_2bebeb18f822.slice/crio-32c2ee7cd9db2c0f787fdf81dd5b2b29d0eea4ac6eed8f62a4da61fd73acbb8c WatchSource:0}: Error finding container 32c2ee7cd9db2c0f787fdf81dd5b2b29d0eea4ac6eed8f62a4da61fd73acbb8c: Status 404 returned error can't find the container with id 32c2ee7cd9db2c0f787fdf81dd5b2b29d0eea4ac6eed8f62a4da61fd73acbb8c Oct 30 04:04:54.745467 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:54.745439 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:04:55.583902 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:55.583858 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" event={"ID":"b5cff3e2-50cd-4110-bf7a-2bebeb18f822","Type":"ContainerStarted","Data":"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b"} Oct 30 04:04:55.584315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:55.583927 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" event={"ID":"b5cff3e2-50cd-4110-bf7a-2bebeb18f822","Type":"ContainerStarted","Data":"32c2ee7cd9db2c0f787fdf81dd5b2b29d0eea4ac6eed8f62a4da61fd73acbb8c"} Oct 30 04:04:55.584315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:55.584063 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:04:55.627677 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:04:55.627623 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" podStartSLOduration=1.261138409 podStartE2EDuration="1.627609687s" totalImagesPullingTime="366.471278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:55.624566918 +0000 UTC m=+3723.609686805" watchObservedRunningTime="2026-10-30 04:04:55.627609687 +0000 UTC m=+3723.612729575" Oct 30 04:05:01.596654 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:01.596622 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:05:04.958783 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:04.958704 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 04:05:04.959182 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:04.958915 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" podUID="96bc333c-d0dd-4969-9d87-5836d80358ee" containerName="maas-api" containerID="cri-o://db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7" gracePeriod=30 Oct 30 04:05:05.256610 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.256578 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 04:05:05.328186 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.328150 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\") pod \"96bc333c-d0dd-4969-9d87-5836d80358ee\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " Oct 30 04:05:05.328397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.328250 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\" (UniqueName: \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\") pod \"96bc333c-d0dd-4969-9d87-5836d80358ee\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " Oct 30 04:05:05.328652 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.328606 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca" pod "96bc333c-d0dd-4969-9d87-5836d80358ee" (UID: "96bc333c-d0dd-4969-9d87-5836d80358ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:05.328928 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.328903 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\") pod \"96bc333c-d0dd-4969-9d87-5836d80358ee\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " Oct 30 04:05:05.329198 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.329178 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\") pod \"96bc333c-d0dd-4969-9d87-5836d80358ee\" (UID: \"96bc333c-d0dd-4969-9d87-5836d80358ee\") " Oct 30 04:05:05.329913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.329884 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc333c-d0dd-4969-9d87-5836d80358ee-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:05.332156 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.332107 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls" pod "96bc333c-d0dd-4969-9d87-5836d80358ee" (UID: "96bc333c-d0dd-4969-9d87-5836d80358ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:05.332763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.332729 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg" pod "96bc333c-d0dd-4969-9d87-5836d80358ee" (UID: "96bc333c-d0dd-4969-9d87-5836d80358ee"). InnerVolumeSpecName "kube-api-access-rqvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:05.333108 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.333083 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs" pod "96bc333c-d0dd-4969-9d87-5836d80358ee" (UID: "96bc333c-d0dd-4969-9d87-5836d80358ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:05.431607 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.431559 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:05.431607 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.431590 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc333c-d0dd-4969-9d87-5836d80358ee-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:05.431607 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.431601 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvlg\" (UniqueName: \"kubernetes.io/projected/96bc333c-d0dd-4969-9d87-5836d80358ee-kube-api-access-rqvlg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:05.651243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.651162 4560 generic.go:356] "Generic (PLEG): container finished" podID="96bc333c-d0dd-4969-9d87-5836d80358ee" containerID="db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7" exitCode=0 Oct 30 04:05:05.651243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.651212 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" Oct 30 04:05:05.651243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.651222 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" event={"ID":"96bc333c-d0dd-4969-9d87-5836d80358ee","Type":"ContainerDied","Data":"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7"} Oct 30 04:05:05.651550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.651262 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk" event={"ID":"96bc333c-d0dd-4969-9d87-5836d80358ee","Type":"ContainerDied","Data":"0a86d71e00a8c419602743f3bd332951260779dcad1386ceca14d72148e19503"} Oct 30 04:05:05.651550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.651283 4560 scope.go:122] "RemoveContainer" containerID="db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7" Oct 30 04:05:05.662387 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.662361 4560 scope.go:122] "RemoveContainer" containerID="db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7" Oct 30 04:05:05.662668 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:05.662647 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7\": container with ID starting with db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7 not found: ID does not exist" containerID="db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7" Oct 30 04:05:05.662732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.662675 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7"} err="failed to get container status \"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7\": rpc error: code = NotFound desc = could not find container \"db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7\": container with ID starting with db883c8a03fb64520410797f91368dc196eb8e2beab189b0cbc3a8d755bdece7 not found: ID does not exist" Oct 30 04:05:05.705144 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.705113 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 04:05:05.708383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:05.708352 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-441461-788748b7-hmhdk"] Oct 30 04:05:06.153103 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:06.153067 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bc333c-d0dd-4969-9d87-5836d80358ee" path="/var/lib/kubelet/pods/96bc333c-d0dd-4969-9d87-5836d80358ee/volumes" Oct 30 04:05:11.751318 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:11.751279 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:05:17.809258 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:17.809223 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 04:05:17.846152 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:17.846103 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 04:05:20.774700 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:20.774657 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:05:22.809675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:22.809620 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" podUID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" containerName="payload-processing" containerID="cri-o://fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6" gracePeriod=25 Oct 30 04:05:22.846899 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:22.846859 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" podUID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" containerName="payload-pre-processing" containerID="cri-o://234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86" gracePeriod=25 Oct 30 04:05:23.120811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.120782 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 04:05:23.152161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.152135 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 04:05:23.271437 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.271396 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\" (UniqueName: \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\") pod \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " Oct 30 04:05:23.271632 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.271446 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\") pod \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " Oct 30 04:05:23.271632 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.271604 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\" (UniqueName: \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\") pod \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\" (UID: \"85978bcc-40fa-4f49-b594-e6c5061b4ae1\") " Oct 30 04:05:23.271731 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.271662 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\") pod \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\" (UID: \"c1b0fe91-c609-4cbb-a99e-165aaf07a294\") " Oct 30 04:05:23.271933 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.271900 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume" pod "85978bcc-40fa-4f49-b594-e6c5061b4ae1" (UID: "85978bcc-40fa-4f49-b594-e6c5061b4ae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:23.272286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.272253 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume" pod "c1b0fe91-c609-4cbb-a99e-165aaf07a294" (UID: "c1b0fe91-c609-4cbb-a99e-165aaf07a294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:23.272467 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.272443 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85978bcc-40fa-4f49-b594-e6c5061b4ae1-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:23.273859 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.273833 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx" pod "85978bcc-40fa-4f49-b594-e6c5061b4ae1" (UID: "85978bcc-40fa-4f49-b594-e6c5061b4ae1"). InnerVolumeSpecName "kube-api-access-t2qsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:23.273958 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.273837 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x" pod "c1b0fe91-c609-4cbb-a99e-165aaf07a294" (UID: "c1b0fe91-c609-4cbb-a99e-165aaf07a294"). InnerVolumeSpecName "kube-api-access-xn44x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:23.373683 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.373597 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qsx\" (UniqueName: \"kubernetes.io/projected/85978bcc-40fa-4f49-b594-e6c5061b4ae1-kube-api-access-t2qsx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:23.373683 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.373627 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0fe91-c609-4cbb-a99e-165aaf07a294-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:23.373683 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.373637 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn44x\" (UniqueName: \"kubernetes.io/projected/c1b0fe91-c609-4cbb-a99e-165aaf07a294-kube-api-access-xn44x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:23.767785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.767748 4560 generic.go:356] "Generic (PLEG): container finished" podID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" containerID="fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6" exitCode=0 Oct 30 04:05:23.767966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.767812 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" Oct 30 04:05:23.767966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.767826 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" event={"ID":"c1b0fe91-c609-4cbb-a99e-165aaf07a294","Type":"ContainerDied","Data":"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6"} Oct 30 04:05:23.767966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.767922 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m" event={"ID":"c1b0fe91-c609-4cbb-a99e-165aaf07a294","Type":"ContainerDied","Data":"8ea09928c663e2ebebe9cd8ae9da9e393403480f0457f5691bea82c727c09339"} Oct 30 04:05:23.767966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.767955 4560 scope.go:122] "RemoveContainer" containerID="fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6" Oct 30 04:05:23.769134 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.769113 4560 generic.go:356] "Generic (PLEG): container finished" podID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" containerID="234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86" exitCode=0 Oct 30 04:05:23.769206 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.769180 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" Oct 30 04:05:23.769255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.769194 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" event={"ID":"85978bcc-40fa-4f49-b594-e6c5061b4ae1","Type":"ContainerDied","Data":"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86"} Oct 30 04:05:23.769255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.769243 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk" event={"ID":"85978bcc-40fa-4f49-b594-e6c5061b4ae1","Type":"ContainerDied","Data":"b88b87e00f68453df496aaed181a79078c55d64caca13c220c157bb84e485067"} Oct 30 04:05:23.778346 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.778318 4560 scope.go:122] "RemoveContainer" containerID="fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6" Oct 30 04:05:23.778733 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:23.778710 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6\": container with ID starting with fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6 not found: ID does not exist" containerID="fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6" Oct 30 04:05:23.778883 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.778742 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6"} err="failed to get container status \"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6\": rpc error: code = NotFound desc = could not find container \"fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6\": container with ID starting with fc7f146f52857ceca5974f40a02f23fdac325723410ec54874daeb3c7e4917e6 not found: ID does not exist" Oct 30 04:05:23.778883 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.778764 4560 scope.go:122] "RemoveContainer" containerID="234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86" Oct 30 04:05:23.788235 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.788217 4560 scope.go:122] "RemoveContainer" containerID="234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86" Oct 30 04:05:23.788533 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:23.788490 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86\": container with ID starting with 234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86 not found: ID does not exist" containerID="234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86" Oct 30 04:05:23.789888 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.789863 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86"} err="failed to get container status \"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86\": rpc error: code = NotFound desc = could not find container \"234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86\": container with ID starting with 234371ca4ca189ac136bc3e0df43697cda5b3854aa953044472364d4ea5eef86 not found: ID does not exist" Oct 30 04:05:23.821104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.821076 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 04:05:23.833094 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.833069 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-81afe3-f7ffcd9f9-62qvk"] Oct 30 04:05:23.857214 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.857188 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 04:05:23.859944 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:23.859919 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-81afe3-7c44c5f4b6-2mk6m"] Oct 30 04:05:24.153549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:24.153459 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" path="/var/lib/kubelet/pods/85978bcc-40fa-4f49-b594-e6c5061b4ae1/volumes" Oct 30 04:05:24.154011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:24.153993 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" path="/var/lib/kubelet/pods/c1b0fe91-c609-4cbb-a99e-165aaf07a294/volumes" Oct 30 04:05:34.140445 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:34.140404 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:05:35.797406 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:35.797356 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" podUID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" containerName="main" containerID="cri-o://3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a" gracePeriod=45 Oct 30 04:05:36.081435 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.081402 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:05:36.120681 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120647 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120715 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120797 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120831 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.120877 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120873 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.121127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.120931 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\" (UniqueName: \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\") pod \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\" (UID: \"ac924b5e-268a-45b6-bf71-3d7b0dde8491\") " Oct 30 04:05:36.121127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121064 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:36.121127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121094 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:36.121261 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121200 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:36.121377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121357 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.121377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121376 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.121614 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.121385 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.123215 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.123186 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:36.123356 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.123330 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:36.124655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.124626 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974" pod "ac924b5e-268a-45b6-bf71-3d7b0dde8491" (UID: "ac924b5e-268a-45b6-bf71-3d7b0dde8491"). InnerVolumeSpecName "kube-api-access-j7974". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:36.222438 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.222409 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac924b5e-268a-45b6-bf71-3d7b0dde8491-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.222438 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.222436 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7974\" (UniqueName: \"kubernetes.io/projected/ac924b5e-268a-45b6-bf71-3d7b0dde8491-kube-api-access-j7974\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.222661 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.222446 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac924b5e-268a-45b6-bf71-3d7b0dde8491-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:36.872504 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.872465 4560 generic.go:356] "Generic (PLEG): container finished" podID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" containerID="3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a" exitCode=0 Oct 30 04:05:36.873085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.872625 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" event={"ID":"ac924b5e-268a-45b6-bf71-3d7b0dde8491","Type":"ContainerDied","Data":"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a"} Oct 30 04:05:36.873085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.872663 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" event={"ID":"ac924b5e-268a-45b6-bf71-3d7b0dde8491","Type":"ContainerDied","Data":"30a34a68f25a4cbdb5a698341a0216e45946bb52cca990ed26ec46d52cc55842"} Oct 30 04:05:36.873085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.872688 4560 scope.go:122] "RemoveContainer" containerID="3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a" Oct 30 04:05:36.873085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.872859 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75" Oct 30 04:05:36.885819 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.885790 4560 scope.go:122] "RemoveContainer" containerID="3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a" Oct 30 04:05:36.886151 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:36.886125 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a\": container with ID starting with 3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a not found: ID does not exist" containerID="3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a" Oct 30 04:05:36.886250 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.886160 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a"} err="failed to get container status \"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a\": rpc error: code = NotFound desc = could not find container \"3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a\": container with ID starting with 3c2e8ff0df46408570f2fe02fb31254f2b894077fa990234fc38cfd5d035287a not found: ID does not exist" Oct 30 04:05:36.943059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.943024 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:05:36.953247 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:36.953207 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-da4553/sub-test-model-da4553-kserve-5f87b865d9-6nj75"] Oct 30 04:05:38.156247 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:38.156213 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" path="/var/lib/kubelet/pods/ac924b5e-268a-45b6-bf71-3d7b0dde8491/volumes" Oct 30 04:05:41.750360 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:41.750321 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:05:44.476835 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.476799 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477434 4560 cpu_manager.go:414] "containerMap: removing container" podUID="96bc333c-d0dd-4969-9d87-5836d80358ee" containerName="maas-api" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477449 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc333c-d0dd-4969-9d87-5836d80358ee" containerName="maas-api" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477460 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" containerName="main" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477466 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" containerName="main" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477487 4560 cpu_manager.go:414] "containerMap: removing container" podUID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" containerName="payload-pre-processing" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477491 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" containerName="payload-pre-processing" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477526 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" containerName="payload-processing" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477534 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" containerName="payload-processing" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477684 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc333c-d0dd-4969-9d87-5836d80358ee" containerName="maas-api" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477698 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac924b5e-268a-45b6-bf71-3d7b0dde8491" containerName="main" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477714 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="85978bcc-40fa-4f49-b594-e6c5061b4ae1" containerName="payload-pre-processing" Oct 30 04:05:44.479287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.477729 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b0fe91-c609-4cbb-a99e-165aaf07a294" containerName="payload-processing" Oct 30 04:05:44.481293 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.481275 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.502616 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.502568 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:05:44.512323 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.512296 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-dd81fd-dockercfg-hpfx2\"" Oct 30 04:05:44.512323 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.512298 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-dd81fd\"" Oct 30 04:05:44.588343 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.588307 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:05:44.594858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.594830 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.604376 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.604344 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:05:44.626021 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.625979 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frl2r\" (UniqueName: \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.626255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.626235 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.727871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.727774 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.727871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.727840 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rz6\" (UniqueName: \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.728115 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.727914 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frl2r\" (UniqueName: \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.728115 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.728031 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.728673 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.728641 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.749011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.748983 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frl2r\" (UniqueName: \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\") pod \"payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.797340 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.797307 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:44.829809 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.829768 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rz6\" (UniqueName: \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.829964 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.829904 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.830660 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.830639 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.852007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.851121 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rz6\" (UniqueName: \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\") pod \"payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:44.918181 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:44.918127 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:45.105968 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.105933 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:05:45.106290 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:05:45.106267 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6057a5a_55ad_4196_9b6d_7c881c04f59f.slice/crio-e63d78f0bb3fc3296202fe8f05610838d0da2e72dc26f0fcad1696299f54fcab WatchSource:0}: Error finding container e63d78f0bb3fc3296202fe8f05610838d0da2e72dc26f0fcad1696299f54fcab: Status 404 returned error can't find the container with id e63d78f0bb3fc3296202fe8f05610838d0da2e72dc26f0fcad1696299f54fcab Oct 30 04:05:45.234338 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.234274 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:05:45.236068 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:05:45.235203 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd68ece_1b4d_4003_b82c_5064e7c3458c.slice/crio-f18c674d90e18618179fa94afd6a32926be6642d82e858bf493bb110f18dbccb WatchSource:0}: Error finding container f18c674d90e18618179fa94afd6a32926be6642d82e858bf493bb110f18dbccb: Status 404 returned error can't find the container with id f18c674d90e18618179fa94afd6a32926be6642d82e858bf493bb110f18dbccb Oct 30 04:05:45.935254 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.935213 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" event={"ID":"7cd68ece-1b4d-4003-b82c-5064e7c3458c","Type":"ContainerStarted","Data":"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b"} Oct 30 04:05:45.935751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.935272 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" event={"ID":"7cd68ece-1b4d-4003-b82c-5064e7c3458c","Type":"ContainerStarted","Data":"f18c674d90e18618179fa94afd6a32926be6642d82e858bf493bb110f18dbccb"} Oct 30 04:05:45.935751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.935310 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:45.936919 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.936895 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" event={"ID":"c6057a5a-55ad-4196-9b6d-7c881c04f59f","Type":"ContainerStarted","Data":"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8"} Oct 30 04:05:45.937054 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.936936 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" event={"ID":"c6057a5a-55ad-4196-9b6d-7c881c04f59f","Type":"ContainerStarted","Data":"e63d78f0bb3fc3296202fe8f05610838d0da2e72dc26f0fcad1696299f54fcab"} Oct 30 04:05:45.937054 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.937018 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:45.979034 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:45.978981 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" podStartSLOduration=1.9789631650000001 podStartE2EDuration="1.978963165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:45.976753949 +0000 UTC m=+3773.961873836" watchObservedRunningTime="2026-10-30 04:05:45.978963165 +0000 UTC m=+3773.964083054" Oct 30 04:05:46.021952 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:46.021891 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" podStartSLOduration=2.02187333 podStartE2EDuration="2.02187333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:46.02096565 +0000 UTC m=+3774.006085540" watchObservedRunningTime="2026-10-30 04:05:46.02187333 +0000 UTC m=+3774.006993221" Oct 30 04:05:48.749624 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:48.749588 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 04:05:48.826288 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:48.826234 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 04:05:49.165242 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.165157 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" containerName="main" containerID="cri-o://2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49" gracePeriod=45 Oct 30 04:05:49.474940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.474911 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:05:49.591992 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.591959 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592153 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592043 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\" (UniqueName: \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592153 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592146 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\" (UniqueName: \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592176 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592207 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592268 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.592468 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592424 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.592563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592442 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\") pod \"30801a51-1c6b-40ee-9540-4c94d37769de\" (UID: \"30801a51-1c6b-40ee-9540-4c94d37769de\") " Oct 30 04:05:49.592563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592543 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.592989 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592962 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.592989 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592985 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.593166 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.592996 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.594398 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.594378 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:49.594482 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.594429 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.594698 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.594681 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq" pod "30801a51-1c6b-40ee-9540-4c94d37769de" (UID: "30801a51-1c6b-40ee-9540-4c94d37769de"). InnerVolumeSpecName "kube-api-access-drgnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:49.694265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.694170 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30801a51-1c6b-40ee-9540-4c94d37769de-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.694265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.694211 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30801a51-1c6b-40ee-9540-4c94d37769de-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.694265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.694226 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgnq\" (UniqueName: \"kubernetes.io/projected/30801a51-1c6b-40ee-9540-4c94d37769de-kube-api-access-drgnq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.964126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.964091 4560 generic.go:356] "Generic (PLEG): container finished" podID="30801a51-1c6b-40ee-9540-4c94d37769de" containerID="2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49" exitCode=0 Oct 30 04:05:49.964675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.964131 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" event={"ID":"30801a51-1c6b-40ee-9540-4c94d37769de","Type":"ContainerDied","Data":"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49"} Oct 30 04:05:49.964675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.964172 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" Oct 30 04:05:49.964675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.964192 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" event={"ID":"30801a51-1c6b-40ee-9540-4c94d37769de","Type":"ContainerDied","Data":"7b3becdf2f62571f37e8b839e3ef5259ffdf37c7c559eb4389c4cd4824218e2c"} Oct 30 04:05:49.964675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.964214 4560 scope.go:122] "RemoveContainer" containerID="2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49" Oct 30 04:05:49.975584 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.975555 4560 scope.go:122] "RemoveContainer" containerID="2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49" Oct 30 04:05:49.976036 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:05:49.975959 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49\": container with ID starting with 2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49 not found: ID does not exist" containerID="2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49" Oct 30 04:05:49.976036 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:49.976002 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49"} err="failed to get container status \"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49\": rpc error: code = NotFound desc = could not find container \"2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49\": container with ID starting with 2c763cbff82472b5d9a35cdc2d5ecb49996c998a8e5ba5d6688d2ec08c919d49 not found: ID does not exist" Oct 30 04:05:50.017946 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:50.017912 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:05:50.023137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:50.023114 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7"] Oct 30 04:05:50.156393 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:50.156353 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" path="/var/lib/kubelet/pods/30801a51-1c6b-40ee-9540-4c94d37769de/volumes" Oct 30 04:05:50.456986 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:50.456946 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-1a1b45/sub-test-model-1a1b45-kserve-6f77ddd85c-f4qd7" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:05:52.809882 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:52.809840 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 04:05:52.810377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:52.810149 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" podUID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" containerName="maas-api" containerID="cri-o://6239449d142007b10d86864b33fb8b3c9267d2be1a07c8d2e21c7639d3ec8e00" gracePeriod=30 Oct 30 04:05:52.987920 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:52.987888 4560 generic.go:356] "Generic (PLEG): container finished" podID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" containerID="6239449d142007b10d86864b33fb8b3c9267d2be1a07c8d2e21c7639d3ec8e00" exitCode=0 Oct 30 04:05:52.988080 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:52.987972 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" event={"ID":"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99","Type":"ContainerDied","Data":"6239449d142007b10d86864b33fb8b3c9267d2be1a07c8d2e21c7639d3ec8e00"} Oct 30 04:05:53.120153 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.120119 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 04:05:53.210600 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.199989 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 04:05:53.210600 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.200234 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" podUID="eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" containerName="maas-api" containerID="cri-o://c7bd0b6ab79327704aa52103bf4ec05c344b9ba5e4251e1f9e6e3dc42ad04776" gracePeriod=30 Oct 30 04:05:53.240909 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.240848 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\") pod \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " Oct 30 04:05:53.240998 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.240928 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\") pod \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " Oct 30 04:05:53.241053 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.241011 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\") pod \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " Oct 30 04:05:53.241053 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.241043 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\" (UniqueName: \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\") pod \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\" (UID: \"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99\") " Oct 30 04:05:53.241960 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.241868 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca" pod "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" (UID: "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:53.247661 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.247627 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs" pod "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" (UID: "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:53.248144 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.248086 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx" pod "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" (UID: "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99"). InnerVolumeSpecName "kube-api-access-n2tsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:53.248465 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.248426 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls" pod "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" (UID: "ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:53.342451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.342367 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.342451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.342400 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2tsx\" (UniqueName: \"kubernetes.io/projected/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-kube-api-access-n2tsx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.342451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.342411 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.342451 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.342420 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.460309 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.460282 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 04:05:53.478564 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.478534 4560 scope.go:122] "RemoveContainer" containerID="c7bd0b6ab79327704aa52103bf4ec05c344b9ba5e4251e1f9e6e3dc42ad04776" Oct 30 04:05:53.492111 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.492033 4560 scope.go:122] "RemoveContainer" containerID="6239449d142007b10d86864b33fb8b3c9267d2be1a07c8d2e21c7639d3ec8e00" Oct 30 04:05:53.544719 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.544684 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\" (UniqueName: \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\") pod \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " Oct 30 04:05:53.544900 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.544778 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\") pod \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " Oct 30 04:05:53.544900 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.544819 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\") pod \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " Oct 30 04:05:53.545007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.544899 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\") pod \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\" (UID: \"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21\") " Oct 30 04:05:53.545440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.545409 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca" pod "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" (UID: "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:53.559570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.547956 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls" pod "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" (UID: "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:53.559570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.554028 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs" pod "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" (UID: "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:53.559570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.554086 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv" pod "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" (UID: "eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21"). InnerVolumeSpecName "kube-api-access-flkcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:53.646550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.646481 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkcv\" (UniqueName: \"kubernetes.io/projected/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-kube-api-access-flkcv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.646550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.646540 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.646550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.646557 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.646796 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.646569 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:05:53.750143 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.750099 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" containerID="cri-o://e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c" gracePeriod=25 Oct 30 04:05:53.826736 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.826694 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" containerID="cri-o://4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741" gracePeriod=25 Oct 30 04:05:53.993939 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.993897 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" event={"ID":"ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99","Type":"ContainerDied","Data":"82025d3ca8b97342f4503f64e78d0e9cecc0a06727baef9180f73d0369e8d7a1"} Oct 30 04:05:53.994160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.993977 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" event={"ID":"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21","Type":"ContainerDied","Data":"c7bd0b6ab79327704aa52103bf4ec05c344b9ba5e4251e1f9e6e3dc42ad04776"} Oct 30 04:05:53.994160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.993985 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" Oct 30 04:05:53.994160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.993999 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr" Oct 30 04:05:53.994160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:53.994004 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr" event={"ID":"eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21","Type":"ContainerDied","Data":"3d8a159e1b545e5d8963271fd0c6831557e5dfbe48e370ee00c90cd8085ffa62"} Oct 30 04:05:54.110813 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:54.110758 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 04:05:54.119059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:54.119026 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9d8692-5cb798bf47-dnzcr"] Oct 30 04:05:54.157478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:54.157442 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" path="/var/lib/kubelet/pods/ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99/volumes" Oct 30 04:05:54.159830 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:54.159800 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 04:05:54.164387 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:54.164361 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-81afe3-65b7757f9b-sqgmr"] Oct 30 04:05:55.166691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:55.166653 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:05:55.167121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:55.166660 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:05:56.153039 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:56.152956 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" path="/var/lib/kubelet/pods/eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21/volumes" Oct 30 04:05:56.946676 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:56.946642 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:05:56.947066 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:56.946729 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:05:57.362681 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.362645 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:05:57.363287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363273 4560 cpu_manager.go:414] "containerMap: removing container" podUID="eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" containerName="maas-api" Oct 30 04:05:57.363335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363288 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" containerName="maas-api" Oct 30 04:05:57.363335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363305 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" containerName="maas-api" Oct 30 04:05:57.363335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363310 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" containerName="maas-api" Oct 30 04:05:57.363335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363327 4560 cpu_manager.go:414] "containerMap: removing container" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" containerName="main" Oct 30 04:05:57.363335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363332 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" containerName="main" Oct 30 04:05:57.363567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363423 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="30801a51-1c6b-40ee-9540-4c94d37769de" containerName="main" Oct 30 04:05:57.363567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363439 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39b4ec-77a8-4d8b-b7b8-606c8ffb6a21" containerName="maas-api" Oct 30 04:05:57.363567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.363446 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4f3ca2-ae0c-4fe6-8fab-18df4f62ab99" containerName="maas-api" Oct 30 04:05:57.368809 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.368790 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.386782 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.386755 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e1a0c2-openshift-default-dockercfg-njtkh\"" Oct 30 04:05:57.389852 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.389823 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:05:57.499917 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.499870 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500096 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.499924 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500096 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500055 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500096 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500177 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500240 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500218 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500243 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500266 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.500429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.500414 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbwq\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602179 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602144 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602179 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602185 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602458 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602242 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602644 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602613 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602644 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602633 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602865 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602673 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602865 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602704 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602865 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602710 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.602865 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602737 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.603107 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.602906 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbwq\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.603107 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.603008 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.603107 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.603072 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.603380 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.603359 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.603708 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.603652 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.605298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.605270 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.606396 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.606375 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.624420 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.624358 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbwq\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.624867 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.624845 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:57.682767 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:57.682733 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:58.020049 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:58.020017 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:05:58.021066 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:05:58.021033 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6ccbee_83a6_4bad_a865_21018fa6e9fa.slice/crio-cff8e02d9e402402787d46ed21d84717a7f62ea9b84c6bba88fac3469cbe8cac WatchSource:0}: Error finding container cff8e02d9e402402787d46ed21d84717a7f62ea9b84c6bba88fac3469cbe8cac: Status 404 returned error can't find the container with id cff8e02d9e402402787d46ed21d84717a7f62ea9b84c6bba88fac3469cbe8cac Oct 30 04:05:58.024059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:58.024024 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:05:58.024134 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:58.024107 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:05:58.024170 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:58.024148 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:05:59.028067 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.028028 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" event={"ID":"7e6ccbee-83a6-4bad-a865-21018fa6e9fa","Type":"ContainerStarted","Data":"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b"} Oct 30 04:05:59.028494 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.028092 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" event={"ID":"7e6ccbee-83a6-4bad-a865-21018fa6e9fa","Type":"ContainerStarted","Data":"cff8e02d9e402402787d46ed21d84717a7f62ea9b84c6bba88fac3469cbe8cac"} Oct 30 04:05:59.108738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.108685 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podStartSLOduration=2.108670307 podStartE2EDuration="2.108670307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:59.107745787 +0000 UTC m=+3787.092865689" watchObservedRunningTime="2026-10-30 04:05:59.108670307 +0000 UTC m=+3787.093790194" Oct 30 04:05:59.683133 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.683094 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:05:59.684707 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.684680 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:05:59.684810 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:05:59.684732 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:00.683115 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:00.683076 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:06:00.683498 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:00.683130 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:01.684113 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:01.684078 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:06:01.684476 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:01.684145 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:02.684232 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:02.684192 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:06:02.684732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:02.684264 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:03.684106 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:03.684066 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:06:03.684312 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:03.684124 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:04.075455 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:04.075417 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 04:06:04.108858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:04.108822 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 04:06:04.683329 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:04.683284 4560 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 04:06:04.683560 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:04.683374 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 04:06:05.167013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:05.166973 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:05.167446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:05.166973 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:05.689110 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:05.689079 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:06:05.689396 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:05.689371 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:06:05.690047 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:05.690025 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:06:08.079017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.078975 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 04:06:08.079512 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.079267 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" podUID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" containerName="maas-api" containerID="cri-o://73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438" gracePeriod=30 Oct 30 04:06:08.375052 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.375022 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 04:06:08.463199 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463167 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\" (UniqueName: \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\") pod \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " Oct 30 04:06:08.463408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463254 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\") pod \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " Oct 30 04:06:08.463408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463323 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\") pod \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " Oct 30 04:06:08.463408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463384 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\") pod \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\" (UID: \"2c984fe5-943a-403a-bd4f-8e467a9c7bb8\") " Oct 30 04:06:08.463763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463727 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca" pod "2c984fe5-943a-403a-bd4f-8e467a9c7bb8" (UID: "2c984fe5-943a-403a-bd4f-8e467a9c7bb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:08.463918 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.463896 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:08.465496 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.465466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk" pod "2c984fe5-943a-403a-bd4f-8e467a9c7bb8" (UID: "2c984fe5-943a-403a-bd4f-8e467a9c7bb8"). InnerVolumeSpecName "kube-api-access-kjnqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:08.465978 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.465937 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls" pod "2c984fe5-943a-403a-bd4f-8e467a9c7bb8" (UID: "2c984fe5-943a-403a-bd4f-8e467a9c7bb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:08.466083 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.465990 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs" pod "2c984fe5-943a-403a-bd4f-8e467a9c7bb8" (UID: "2c984fe5-943a-403a-bd4f-8e467a9c7bb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:08.565548 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.565473 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:08.565548 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.565509 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:08.565767 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.565556 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjnqk\" (UniqueName: \"kubernetes.io/projected/2c984fe5-943a-403a-bd4f-8e467a9c7bb8-kube-api-access-kjnqk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:08.637609 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.637504 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 04:06:08.637800 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.637763 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" podUID="e9856027-c513-4c98-b2af-f8834eede869" containerName="maas-api" containerID="cri-o://316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e" gracePeriod=30 Oct 30 04:06:08.914447 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.914410 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 04:06:08.970761 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.970726 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\") pod \"e9856027-c513-4c98-b2af-f8834eede869\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " Oct 30 04:06:08.970954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.970835 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\" (UniqueName: \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\") pod \"e9856027-c513-4c98-b2af-f8834eede869\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " Oct 30 04:06:08.970954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.970911 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\") pod \"e9856027-c513-4c98-b2af-f8834eede869\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " Oct 30 04:06:08.971044 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.970993 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\") pod \"e9856027-c513-4c98-b2af-f8834eede869\" (UID: \"e9856027-c513-4c98-b2af-f8834eede869\") " Oct 30 04:06:08.972254 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.972221 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca" pod "e9856027-c513-4c98-b2af-f8834eede869" (UID: "e9856027-c513-4c98-b2af-f8834eede869"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:08.972442 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.972421 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9856027-c513-4c98-b2af-f8834eede869-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:08.973988 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.973947 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls" pod "e9856027-c513-4c98-b2af-f8834eede869" (UID: "e9856027-c513-4c98-b2af-f8834eede869"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:08.973988 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.973959 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs" pod "e9856027-c513-4c98-b2af-f8834eede869" (UID: "e9856027-c513-4c98-b2af-f8834eede869"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:08.974773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:08.974746 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc" pod "e9856027-c513-4c98-b2af-f8834eede869" (UID: "e9856027-c513-4c98-b2af-f8834eede869"). InnerVolumeSpecName "kube-api-access-g2zpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:09.074335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.074292 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:09.074335 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.074335 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zpc\" (UniqueName: \"kubernetes.io/projected/e9856027-c513-4c98-b2af-f8834eede869-kube-api-access-g2zpc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:09.074596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.074351 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9856027-c513-4c98-b2af-f8834eede869-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:09.076486 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.076451 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" containerID="cri-o://cd4e538d322c4208105fb3fb5e5a7a8601f1a3aba5f90093e91b4cd5ddd3bc39" gracePeriod=25 Oct 30 04:06:09.109181 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.109135 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" containerID="cri-o://6c2561d24d74ddf0a2d9bd2b27d5256c65702889b5ab2c4f8a52a54cb3eeb3dc" gracePeriod=25 Oct 30 04:06:09.160049 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.159968 4560 generic.go:356] "Generic (PLEG): container finished" podID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" containerID="73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438" exitCode=0 Oct 30 04:06:09.160049 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.160026 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" Oct 30 04:06:09.160286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.160041 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" event={"ID":"2c984fe5-943a-403a-bd4f-8e467a9c7bb8","Type":"ContainerDied","Data":"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438"} Oct 30 04:06:09.160286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.160104 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j" event={"ID":"2c984fe5-943a-403a-bd4f-8e467a9c7bb8","Type":"ContainerDied","Data":"48148c508dd4a1a8cf86364a3cdc98c238438182e7a38ef9b1cf7fc3e4627110"} Oct 30 04:06:09.160286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.160133 4560 scope.go:122] "RemoveContainer" containerID="73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438" Oct 30 04:06:09.161668 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.161644 4560 generic.go:356] "Generic (PLEG): container finished" podID="e9856027-c513-4c98-b2af-f8834eede869" containerID="316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e" exitCode=0 Oct 30 04:06:09.161782 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.161690 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" Oct 30 04:06:09.161782 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.161692 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" event={"ID":"e9856027-c513-4c98-b2af-f8834eede869","Type":"ContainerDied","Data":"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e"} Oct 30 04:06:09.161886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.161813 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9" event={"ID":"e9856027-c513-4c98-b2af-f8834eede869","Type":"ContainerDied","Data":"e7dd18a38d464ebf401172be1cf2f8bd0251e8137ef428cc02e5d3f5c33272ee"} Oct 30 04:06:09.184286 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.184265 4560 scope.go:122] "RemoveContainer" containerID="73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438" Oct 30 04:06:09.184661 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:09.184641 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438\": container with ID starting with 73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438 not found: ID does not exist" containerID="73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438" Oct 30 04:06:09.184719 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.184668 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438"} err="failed to get container status \"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438\": rpc error: code = NotFound desc = could not find container \"73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438\": container with ID starting with 73f64736f84b8940dbcd12095b4f768a43df5c0a629aa74202de6980831dd438 not found: ID does not exist" Oct 30 04:06:09.184719 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.184686 4560 scope.go:122] "RemoveContainer" containerID="316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e" Oct 30 04:06:09.194098 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.194077 4560 scope.go:122] "RemoveContainer" containerID="316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e" Oct 30 04:06:09.194400 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:09.194378 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e\": container with ID starting with 316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e not found: ID does not exist" containerID="316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e" Oct 30 04:06:09.194476 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.194407 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e"} err="failed to get container status \"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e\": rpc error: code = NotFound desc = could not find container \"316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e\": container with ID starting with 316271c4f56ba82c13b58e592a8858870204fd46105fc810b4b5f25559737b5e not found: ID does not exist" Oct 30 04:06:09.226136 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:09.226101 4560 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c984fe5_943a_403a_bd4f_8e467a9c7bb8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9856027_c513_4c98_b2af_f8834eede869.slice\": RecentStats: unable to find data in memory cache]" Oct 30 04:06:09.237123 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.237094 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 04:06:09.250298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.250269 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689bcf-d844cf467-gtx6j"] Oct 30 04:06:09.280009 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.279975 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 04:06:09.283819 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:09.283794 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93b821-67b8f4d6cf-bvkw9"] Oct 30 04:06:10.152671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:10.152641 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" path="/var/lib/kubelet/pods/2c984fe5-943a-403a-bd4f-8e467a9c7bb8/volumes" Oct 30 04:06:10.153162 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:10.153147 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9856027-c513-4c98-b2af-f8834eede869" path="/var/lib/kubelet/pods/e9856027-c513-4c98-b2af-f8834eede869/volumes" Oct 30 04:06:11.519448 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:11.519410 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:11.519902 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:11.519410 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:11.749489 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:11.749448 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:06:14.129007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.128974 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 04:06:14.159655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.159627 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 04:06:14.196803 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.196767 4560 generic.go:356] "Generic (PLEG): container finished" podID="c765941d-9664-4e4b-b039-2b255c1ade47" containerID="4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741" exitCode=0 Oct 30 04:06:14.196991 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.196832 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" Oct 30 04:06:14.196991 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.196832 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" event={"ID":"c765941d-9664-4e4b-b039-2b255c1ade47","Type":"ContainerDied","Data":"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741"} Oct 30 04:06:14.197109 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.196999 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr" event={"ID":"c765941d-9664-4e4b-b039-2b255c1ade47","Type":"ContainerDied","Data":"3af3d5d5f917f4e1a741b860d91e5027fd7fc7f0b3b32addcade7e0c8ffb00ed"} Oct 30 04:06:14.197109 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.197047 4560 scope.go:122] "RemoveContainer" containerID="4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741" Oct 30 04:06:14.198560 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.198535 4560 generic.go:356] "Generic (PLEG): container finished" podID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerID="e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c" exitCode=0 Oct 30 04:06:14.198677 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.198606 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" Oct 30 04:06:14.198677 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.198619 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" event={"ID":"f29d3c12-21bb-43f6-8bdf-33791fb683e5","Type":"ContainerDied","Data":"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c"} Oct 30 04:06:14.198793 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.198695 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv" event={"ID":"f29d3c12-21bb-43f6-8bdf-33791fb683e5","Type":"ContainerDied","Data":"9c1957dda8adda2b10c2279b2cc117e2329eea0f6fed240c517212b0f5ac4a95"} Oct 30 04:06:14.207848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.207833 4560 scope.go:122] "RemoveContainer" containerID="4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741" Oct 30 04:06:14.208102 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:14.208080 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741\": container with ID starting with 4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741 not found: ID does not exist" containerID="4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741" Oct 30 04:06:14.208176 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.208108 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741"} err="failed to get container status \"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741\": rpc error: code = NotFound desc = could not find container \"4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741\": container with ID starting with 4dccd0708fb60fc11768e47bde4ca6fa57c3047e1a3328d80b48610897bb2741 not found: ID does not exist" Oct 30 04:06:14.208176 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.208124 4560 scope.go:122] "RemoveContainer" containerID="e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c" Oct 30 04:06:14.216961 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.216944 4560 scope.go:122] "RemoveContainer" containerID="e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c" Oct 30 04:06:14.217234 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:14.217211 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c\": container with ID starting with e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c not found: ID does not exist" containerID="e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c" Oct 30 04:06:14.217303 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.217237 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c"} err="failed to get container status \"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c\": rpc error: code = NotFound desc = could not find container \"e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c\": container with ID starting with e37ed9431dd41f06aec0fe2e9f355e39aa0238b3016aa2e8a00b07603272520c not found: ID does not exist" Oct 30 04:06:14.236707 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.236676 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\" (UniqueName: \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\") pod \"c765941d-9664-4e4b-b039-2b255c1ade47\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " Oct 30 04:06:14.236858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.236827 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\") pod \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " Oct 30 04:06:14.236975 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.236953 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\") pod \"c765941d-9664-4e4b-b039-2b255c1ade47\" (UID: \"c765941d-9664-4e4b-b039-2b255c1ade47\") " Oct 30 04:06:14.237106 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.237075 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\" (UniqueName: \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\") pod \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\" (UID: \"f29d3c12-21bb-43f6-8bdf-33791fb683e5\") " Oct 30 04:06:14.237291 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.237265 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume" pod "f29d3c12-21bb-43f6-8bdf-33791fb683e5" (UID: "f29d3c12-21bb-43f6-8bdf-33791fb683e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:14.237394 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.237370 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume" pod "c765941d-9664-4e4b-b039-2b255c1ade47" (UID: "c765941d-9664-4e4b-b039-2b255c1ade47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:14.237745 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.237707 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29d3c12-21bb-43f6-8bdf-33791fb683e5-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:14.237868 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.237751 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c765941d-9664-4e4b-b039-2b255c1ade47-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:14.239061 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.239034 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4" pod "c765941d-9664-4e4b-b039-2b255c1ade47" (UID: "c765941d-9664-4e4b-b039-2b255c1ade47"). InnerVolumeSpecName "kube-api-access-7b5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:14.239878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.239856 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq" pod "f29d3c12-21bb-43f6-8bdf-33791fb683e5" (UID: "f29d3c12-21bb-43f6-8bdf-33791fb683e5"). InnerVolumeSpecName "kube-api-access-lx9bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:14.339317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.339226 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9bq\" (UniqueName: \"kubernetes.io/projected/f29d3c12-21bb-43f6-8bdf-33791fb683e5-kube-api-access-lx9bq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:14.339317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.339259 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5t4\" (UniqueName: \"kubernetes.io/projected/c765941d-9664-4e4b-b039-2b255c1ade47-kube-api-access-7b5t4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:14.550170 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.550137 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 04:06:14.563170 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.563132 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9d8692-5b77b85647-2g4kr"] Oct 30 04:06:14.589639 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.589551 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 04:06:14.596389 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:14.596356 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9d8692-79b5454cb4-fsbrv"] Oct 30 04:06:16.154177 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:16.154135 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" path="/var/lib/kubelet/pods/c765941d-9664-4e4b-b039-2b255c1ade47/volumes" Oct 30 04:06:16.154793 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:16.154774 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" path="/var/lib/kubelet/pods/f29d3c12-21bb-43f6-8bdf-33791fb683e5/volumes" Oct 30 04:06:17.420076 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420036 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:06:17.420894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420869 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" containerName="maas-api" Oct 30 04:06:17.420894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420893 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" containerName="maas-api" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420916 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e9856027-c513-4c98-b2af-f8834eede869" containerName="maas-api" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420923 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9856027-c513-4c98-b2af-f8834eede869" containerName="maas-api" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420971 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.420980 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421001 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" Oct 30 04:06:17.421079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421007 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" Oct 30 04:06:17.421397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421171 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29d3c12-21bb-43f6-8bdf-33791fb683e5" containerName="payload-processing" Oct 30 04:06:17.421397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421191 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c984fe5-943a-403a-bd4f-8e467a9c7bb8" containerName="maas-api" Oct 30 04:06:17.421397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421206 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9856027-c513-4c98-b2af-f8834eede869" containerName="maas-api" Oct 30 04:06:17.421397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.421223 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c765941d-9664-4e4b-b039-2b255c1ade47" containerName="payload-pre-processing" Oct 30 04:06:17.426763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.426738 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.444104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.444069 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-dd81fd-serving-cert\"" Oct 30 04:06:17.447604 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.447572 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:06:17.579986 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.579939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5q2\" (UniqueName: \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.580178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.580000 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.580178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.580086 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.580178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.580136 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.682051 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.681961 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5q2\" (UniqueName: \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.682051 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.682007 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.682051 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.682026 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.682302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.682059 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.682783 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.682753 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.684510 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.684485 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.684638 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.684571 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.704381 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.704346 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5q2\" (UniqueName: \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\") pod \"maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:17.741740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:17.741700 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:18.047566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:18.047538 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:06:18.048203 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:06:18.048172 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dff755_f0bf_49b5_b9bd_5c6fc649154f.slice/crio-0374d29b7fc3473254b203b5b062ba7383dc469a1d420d6795692ae048e5de6b WatchSource:0}: Error finding container 0374d29b7fc3473254b203b5b062ba7383dc469a1d420d6795692ae048e5de6b: Status 404 returned error can't find the container with id 0374d29b7fc3473254b203b5b062ba7383dc469a1d420d6795692ae048e5de6b Oct 30 04:06:18.238349 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:18.238297 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" event={"ID":"85dff755-f0bf-49b5-b9bd-5c6fc649154f","Type":"ContainerStarted","Data":"0374d29b7fc3473254b203b5b062ba7383dc469a1d420d6795692ae048e5de6b"} Oct 30 04:06:19.247812 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:19.247771 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" event={"ID":"85dff755-f0bf-49b5-b9bd-5c6fc649154f","Type":"ContainerStarted","Data":"ca5de28dbacf4de5d751779cbd52c8cd3b5de9d0166df83166efaaef5db49ff3"} Oct 30 04:06:19.248616 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:19.247877 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:19.291392 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:19.291334 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" podStartSLOduration=1.943315734 podStartE2EDuration="2.291318818s" totalImagesPullingTime="348.003084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:19.28975715 +0000 UTC m=+3807.274877038" watchObservedRunningTime="2026-10-30 04:06:19.291318818 +0000 UTC m=+3807.276438705" Oct 30 04:06:21.518819 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:21.518783 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:21.518819 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:21.518814 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:23.654686 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:23.654648 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 04:06:25.260017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:25.259988 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:06:28.232460 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.232420 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 04:06:28.233022 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.232712 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" podUID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" containerName="maas-api" containerID="cri-o://44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c" gracePeriod=30 Oct 30 04:06:28.551337 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.551301 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 04:06:28.622601 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.622562 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\") pod \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " Oct 30 04:06:28.622774 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.622629 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\" (UniqueName: \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\") pod \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " Oct 30 04:06:28.622774 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.622729 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") pod \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " Oct 30 04:06:28.622913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.622858 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\") pod \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\" (UID: \"28f93420-c8bc-44f8-a5f3-d0d4ad165920\") " Oct 30 04:06:28.623013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.622986 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca" pod "28f93420-c8bc-44f8-a5f3-d0d4ad165920" (UID: "28f93420-c8bc-44f8-a5f3-d0d4ad165920"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:28.623427 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.623400 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f93420-c8bc-44f8-a5f3-d0d4ad165920-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.625071 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.624999 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v" pod "28f93420-c8bc-44f8-a5f3-d0d4ad165920" (UID: "28f93420-c8bc-44f8-a5f3-d0d4ad165920"). InnerVolumeSpecName "kube-api-access-svj9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:28.625071 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.625018 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls" pod "28f93420-c8bc-44f8-a5f3-d0d4ad165920" (UID: "28f93420-c8bc-44f8-a5f3-d0d4ad165920"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:28.633954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.633923 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs" pod "28f93420-c8bc-44f8-a5f3-d0d4ad165920" (UID: "28f93420-c8bc-44f8-a5f3-d0d4ad165920"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:28.655164 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.655125 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" containerID="cri-o://c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15" gracePeriod=25 Oct 30 04:06:28.724226 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.724191 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svj9v\" (UniqueName: \"kubernetes.io/projected/28f93420-c8bc-44f8-a5f3-d0d4ad165920-kube-api-access-svj9v\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.724226 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.724223 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.724226 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:28.724234 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f93420-c8bc-44f8-a5f3-d0d4ad165920-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:29.322238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.322046 4560 generic.go:356] "Generic (PLEG): container finished" podID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" containerID="44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c" exitCode=0 Oct 30 04:06:29.322238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.322106 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" Oct 30 04:06:29.322238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.322135 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" event={"ID":"28f93420-c8bc-44f8-a5f3-d0d4ad165920","Type":"ContainerDied","Data":"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c"} Oct 30 04:06:29.322238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.322207 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd" event={"ID":"28f93420-c8bc-44f8-a5f3-d0d4ad165920","Type":"ContainerDied","Data":"e86601f75bd30cc950cae2606be7d70b19e8b276e83da3a9fc253849ca8dc224"} Oct 30 04:06:29.322238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.322237 4560 scope.go:122] "RemoveContainer" containerID="44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c" Oct 30 04:06:29.334010 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.332595 4560 generic.go:356] "Generic (PLEG): container finished" podID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerID="6c2561d24d74ddf0a2d9bd2b27d5256c65702889b5ab2c4f8a52a54cb3eeb3dc" exitCode=0 Oct 30 04:06:29.334010 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.332667 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" event={"ID":"1eb0bb28-a713-4d4b-b5c4-b882416ddea7","Type":"ContainerDied","Data":"6c2561d24d74ddf0a2d9bd2b27d5256c65702889b5ab2c4f8a52a54cb3eeb3dc"} Oct 30 04:06:29.336769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.334640 4560 generic.go:356] "Generic (PLEG): container finished" podID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerID="cd4e538d322c4208105fb3fb5e5a7a8601f1a3aba5f90093e91b4cd5ddd3bc39" exitCode=0 Oct 30 04:06:29.336769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.334715 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" event={"ID":"a47f8638-835e-422b-bb51-2dc58acba6e1","Type":"ContainerDied","Data":"cd4e538d322c4208105fb3fb5e5a7a8601f1a3aba5f90093e91b4cd5ddd3bc39"} Oct 30 04:06:29.350704 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.350606 4560 scope.go:122] "RemoveContainer" containerID="44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c" Oct 30 04:06:29.351027 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:29.350959 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c\": container with ID starting with 44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c not found: ID does not exist" containerID="44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c" Oct 30 04:06:29.351027 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.350998 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c"} err="failed to get container status \"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c\": rpc error: code = NotFound desc = could not find container \"44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c\": container with ID starting with 44bd224e1bd55d990b1261582f0d4a7db5021983b16ca6adb94595e2b4ca317c not found: ID does not exist" Oct 30 04:06:29.390490 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.390453 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 04:06:29.395954 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.395926 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6ad168-6d4f6f565-snskd"] Oct 30 04:06:29.400696 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.400677 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 04:06:29.431689 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.431664 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 04:06:29.431964 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.431946 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\" (UniqueName: \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\") pod \"a47f8638-835e-422b-bb51-2dc58acba6e1\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " Oct 30 04:06:29.432084 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.432072 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\") pod \"a47f8638-835e-422b-bb51-2dc58acba6e1\" (UID: \"a47f8638-835e-422b-bb51-2dc58acba6e1\") " Oct 30 04:06:29.432488 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.432453 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume" pod "a47f8638-835e-422b-bb51-2dc58acba6e1" (UID: "a47f8638-835e-422b-bb51-2dc58acba6e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:29.433922 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.433902 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj" pod "a47f8638-835e-422b-bb51-2dc58acba6e1" (UID: "a47f8638-835e-422b-bb51-2dc58acba6e1"). InnerVolumeSpecName "kube-api-access-wqrzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:29.532919 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.532883 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\" (UniqueName: \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\") pod \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " Oct 30 04:06:29.533086 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.533017 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\") pod \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\" (UID: \"1eb0bb28-a713-4d4b-b5c4-b882416ddea7\") " Oct 30 04:06:29.533398 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.533375 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume" pod "1eb0bb28-a713-4d4b-b5c4-b882416ddea7" (UID: "1eb0bb28-a713-4d4b-b5c4-b882416ddea7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:29.533665 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.533642 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrzj\" (UniqueName: \"kubernetes.io/projected/a47f8638-835e-422b-bb51-2dc58acba6e1-kube-api-access-wqrzj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:29.533746 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.533669 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:29.533746 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.533685 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f8638-835e-422b-bb51-2dc58acba6e1-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:29.535079 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.535056 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9" pod "1eb0bb28-a713-4d4b-b5c4-b882416ddea7" (UID: "1eb0bb28-a713-4d4b-b5c4-b882416ddea7"). InnerVolumeSpecName "kube-api-access-kzjc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:29.634762 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:29.634687 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjc9\" (UniqueName: \"kubernetes.io/projected/1eb0bb28-a713-4d4b-b5c4-b882416ddea7-kube-api-access-kzjc9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:30.156212 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.156178 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" path="/var/lib/kubelet/pods/28f93420-c8bc-44f8-a5f3-d0d4ad165920/volumes" Oct 30 04:06:30.343091 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.343056 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" Oct 30 04:06:30.343569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.343064 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc" event={"ID":"1eb0bb28-a713-4d4b-b5c4-b882416ddea7","Type":"ContainerDied","Data":"9a161f2fc58846b062d56f8559d5c7dacccc6aa9bf9d89d413d90e57e44e0005"} Oct 30 04:06:30.343569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.343211 4560 scope.go:122] "RemoveContainer" containerID="6c2561d24d74ddf0a2d9bd2b27d5256c65702889b5ab2c4f8a52a54cb3eeb3dc" Oct 30 04:06:30.344978 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.344956 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" Oct 30 04:06:30.345110 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.345002 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z" event={"ID":"a47f8638-835e-422b-bb51-2dc58acba6e1","Type":"ContainerDied","Data":"37b6532ecd0f8e95cdf18e90b1b60d08e808c0db5df7de1bb123444514c015cd"} Oct 30 04:06:30.353381 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.353364 4560 scope.go:122] "RemoveContainer" containerID="cd4e538d322c4208105fb3fb5e5a7a8601f1a3aba5f90093e91b4cd5ddd3bc39" Oct 30 04:06:30.400664 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.400633 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 04:06:30.413229 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.413169 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93b821-867684b6c9-rrfgc"] Oct 30 04:06:30.452112 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.452081 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 04:06:30.465413 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:30.465386 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93b821-7f884d5c88-xlj6z"] Oct 30 04:06:32.152645 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:32.152605 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" path="/var/lib/kubelet/pods/1eb0bb28-a713-4d4b-b5c4-b882416ddea7/volumes" Oct 30 04:06:32.153198 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:32.153177 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" path="/var/lib/kubelet/pods/a47f8638-835e-422b-bb51-2dc58acba6e1/volumes" Oct 30 04:06:37.428680 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:37.428585 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:41.749836 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:41.749791 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:06:44.507401 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.507362 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 04:06:44.508211 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.508147 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" podUID="7db61479-cb23-4486-9b05-f99634f64df7" containerName="maas-api" containerID="cri-o://b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2" gracePeriod=30 Oct 30 04:06:44.809556 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.809526 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 04:06:44.939412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.939380 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\" (UniqueName: \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\") pod \"7db61479-cb23-4486-9b05-f99634f64df7\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " Oct 30 04:06:44.939631 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.939506 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\") pod \"7db61479-cb23-4486-9b05-f99634f64df7\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " Oct 30 04:06:44.939631 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.939583 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\") pod \"7db61479-cb23-4486-9b05-f99634f64df7\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " Oct 30 04:06:44.939728 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.939678 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\") pod \"7db61479-cb23-4486-9b05-f99634f64df7\" (UID: \"7db61479-cb23-4486-9b05-f99634f64df7\") " Oct 30 04:06:44.940098 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.940063 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca" pod "7db61479-cb23-4486-9b05-f99634f64df7" (UID: "7db61479-cb23-4486-9b05-f99634f64df7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:44.941714 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.941685 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls" pod "7db61479-cb23-4486-9b05-f99634f64df7" (UID: "7db61479-cb23-4486-9b05-f99634f64df7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:44.941813 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.941727 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897" pod "7db61479-cb23-4486-9b05-f99634f64df7" (UID: "7db61479-cb23-4486-9b05-f99634f64df7"). InnerVolumeSpecName "kube-api-access-98897". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:44.941813 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:44.941795 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs" pod "7db61479-cb23-4486-9b05-f99634f64df7" (UID: "7db61479-cb23-4486-9b05-f99634f64df7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:45.041269 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.041180 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7db61479-cb23-4486-9b05-f99634f64df7-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:45.041269 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.041214 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98897\" (UniqueName: \"kubernetes.io/projected/7db61479-cb23-4486-9b05-f99634f64df7-kube-api-access-98897\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:45.041269 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.041226 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:45.041269 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.041236 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7db61479-cb23-4486-9b05-f99634f64df7-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:45.441832 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.441752 4560 generic.go:356] "Generic (PLEG): container finished" podID="7db61479-cb23-4486-9b05-f99634f64df7" containerID="b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2" exitCode=0 Oct 30 04:06:45.441832 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.441795 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" Oct 30 04:06:45.442023 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.441833 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" event={"ID":"7db61479-cb23-4486-9b05-f99634f64df7","Type":"ContainerDied","Data":"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2"} Oct 30 04:06:45.442023 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.441890 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg" event={"ID":"7db61479-cb23-4486-9b05-f99634f64df7","Type":"ContainerDied","Data":"c00f73fd8840770d0da11dcb5e545d3464edab0a966a8f510a194195d1750fb5"} Oct 30 04:06:45.442023 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.441911 4560 scope.go:122] "RemoveContainer" containerID="b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2" Oct 30 04:06:45.451854 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.451838 4560 scope.go:122] "RemoveContainer" containerID="b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2" Oct 30 04:06:45.452115 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:45.452096 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2\": container with ID starting with b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2 not found: ID does not exist" containerID="b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2" Oct 30 04:06:45.452163 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.452125 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2"} err="failed to get container status \"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2\": rpc error: code = NotFound desc = could not find container \"b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2\": container with ID starting with b0694ac2103b222c8b8c2181d3acdd41113a13e8009427c974d15eb21a5edce2 not found: ID does not exist" Oct 30 04:06:45.496802 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.496764 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 04:06:45.509529 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:45.509482 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-547a78-5757d99c67-6lmzg"] Oct 30 04:06:46.153685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:46.153641 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db61479-cb23-4486-9b05-f99634f64df7" path="/var/lib/kubelet/pods/7db61479-cb23-4486-9b05-f99634f64df7/volumes" Oct 30 04:06:47.428435 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:47.428396 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:48.925958 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.925900 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 04:06:48.988230 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.988193 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\" (UniqueName: \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\") pod \"34f7a842-978b-4921-a056-267eca8e8cab\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " Oct 30 04:06:48.988436 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.988393 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\") pod \"34f7a842-978b-4921-a056-267eca8e8cab\" (UID: \"34f7a842-978b-4921-a056-267eca8e8cab\") " Oct 30 04:06:48.988885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.988859 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume" pod "34f7a842-978b-4921-a056-267eca8e8cab" (UID: "34f7a842-978b-4921-a056-267eca8e8cab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:48.989380 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.989358 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f7a842-978b-4921-a056-267eca8e8cab-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:48.990881 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:48.990851 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92" pod "34f7a842-978b-4921-a056-267eca8e8cab" (UID: "34f7a842-978b-4921-a056-267eca8e8cab"). InnerVolumeSpecName "kube-api-access-r5m92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:49.091095 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.091061 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5m92\" (UniqueName: \"kubernetes.io/projected/34f7a842-978b-4921-a056-267eca8e8cab-kube-api-access-r5m92\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:49.467916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.467881 4560 generic.go:356] "Generic (PLEG): container finished" podID="34f7a842-978b-4921-a056-267eca8e8cab" containerID="c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15" exitCode=0 Oct 30 04:06:49.468121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.467928 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" Oct 30 04:06:49.468121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.467941 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" event={"ID":"34f7a842-978b-4921-a056-267eca8e8cab","Type":"ContainerDied","Data":"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15"} Oct 30 04:06:49.468121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.467982 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl" event={"ID":"34f7a842-978b-4921-a056-267eca8e8cab","Type":"ContainerDied","Data":"53ddc820d5fc4f111be8d96b12238f06c1e5ef706830c395172e88182f5bbebe"} Oct 30 04:06:49.468121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.468001 4560 scope.go:122] "RemoveContainer" containerID="c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15" Oct 30 04:06:49.478330 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.478292 4560 scope.go:122] "RemoveContainer" containerID="c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15" Oct 30 04:06:49.478668 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:49.478647 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15\": container with ID starting with c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15 not found: ID does not exist" containerID="c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15" Oct 30 04:06:49.478743 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.478677 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15"} err="failed to get container status \"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15\": rpc error: code = NotFound desc = could not find container \"c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15\": container with ID starting with c0362385a432372943647ddc274946aec551852c3aa2ba3c28e1de648128fb15 not found: ID does not exist" Oct 30 04:06:49.521843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.521808 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 04:06:49.533953 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:49.533921 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-35f43f-69d8b5d5b7-mjmvl"] Oct 30 04:06:50.152408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:50.152367 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f7a842-978b-4921-a056-267eca8e8cab" path="/var/lib/kubelet/pods/34f7a842-978b-4921-a056-267eca8e8cab/volumes" Oct 30 04:06:55.644870 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:55.644835 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 04:06:55.645378 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:55.645134 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" podUID="087fc4d2-134e-44b9-aa65-7385b9f96b19" containerName="maas-api" containerID="cri-o://30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac" gracePeriod=30 Oct 30 04:06:55.941874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:55.941846 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 04:06:56.081326 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.081284 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\") pod \"087fc4d2-134e-44b9-aa65-7385b9f96b19\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " Oct 30 04:06:56.081507 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.081333 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\") pod \"087fc4d2-134e-44b9-aa65-7385b9f96b19\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " Oct 30 04:06:56.081507 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.081441 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\") pod \"087fc4d2-134e-44b9-aa65-7385b9f96b19\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " Oct 30 04:06:56.081507 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.081500 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\") pod \"087fc4d2-134e-44b9-aa65-7385b9f96b19\" (UID: \"087fc4d2-134e-44b9-aa65-7385b9f96b19\") " Oct 30 04:06:56.081823 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.081796 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca" pod "087fc4d2-134e-44b9-aa65-7385b9f96b19" (UID: "087fc4d2-134e-44b9-aa65-7385b9f96b19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:56.082059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.082034 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087fc4d2-134e-44b9-aa65-7385b9f96b19-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:56.083690 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.083659 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls" pod "087fc4d2-134e-44b9-aa65-7385b9f96b19" (UID: "087fc4d2-134e-44b9-aa65-7385b9f96b19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:56.084127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.084103 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs" pod "087fc4d2-134e-44b9-aa65-7385b9f96b19" (UID: "087fc4d2-134e-44b9-aa65-7385b9f96b19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:56.084212 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.084107 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh" pod "087fc4d2-134e-44b9-aa65-7385b9f96b19" (UID: "087fc4d2-134e-44b9-aa65-7385b9f96b19"). InnerVolumeSpecName "kube-api-access-8p9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:56.183638 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.183556 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/087fc4d2-134e-44b9-aa65-7385b9f96b19-kube-api-access-8p9rh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:56.183638 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.183584 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:56.183638 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.183594 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087fc4d2-134e-44b9-aa65-7385b9f96b19-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:56.522863 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.522828 4560 generic.go:356] "Generic (PLEG): container finished" podID="087fc4d2-134e-44b9-aa65-7385b9f96b19" containerID="30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac" exitCode=0 Oct 30 04:06:56.523057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.522874 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" Oct 30 04:06:56.523057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.522882 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" event={"ID":"087fc4d2-134e-44b9-aa65-7385b9f96b19","Type":"ContainerDied","Data":"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac"} Oct 30 04:06:56.523057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.522927 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw" event={"ID":"087fc4d2-134e-44b9-aa65-7385b9f96b19","Type":"ContainerDied","Data":"009c2c49ae32eb8bc55c61761de8e947c97e92240d657295d5ca8057153f443c"} Oct 30 04:06:56.523057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.522950 4560 scope.go:122] "RemoveContainer" containerID="30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac" Oct 30 04:06:56.533921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.533895 4560 scope.go:122] "RemoveContainer" containerID="30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac" Oct 30 04:06:56.534249 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:06:56.534225 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac\": container with ID starting with 30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac not found: ID does not exist" containerID="30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac" Oct 30 04:06:56.534299 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.534260 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac"} err="failed to get container status \"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac\": rpc error: code = NotFound desc = could not find container \"30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac\": container with ID starting with 30ec2c5df98a4b3864967d1f0024d46afc4054f8c18859bda8b7855a095b75ac not found: ID does not exist" Oct 30 04:06:56.576632 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.576587 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 04:06:56.589456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:56.589423 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-35f43f-565d959f59-p6jdw"] Oct 30 04:06:58.152410 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:58.152374 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087fc4d2-134e-44b9-aa65-7385b9f96b19" path="/var/lib/kubelet/pods/087fc4d2-134e-44b9-aa65-7385b9f96b19/volumes" Oct 30 04:06:59.217280 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.217236 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 04:06:59.226019 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.217485 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" podUID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" containerName="maas-api" containerID="cri-o://a7cc62ac43e12846f507b8519a6ba17a55290285d9ea8d269d0d6fb22721299d" gracePeriod=30 Oct 30 04:06:59.276479 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.276439 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 04:06:59.542099 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.542071 4560 generic.go:356] "Generic (PLEG): container finished" podID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" containerID="a7cc62ac43e12846f507b8519a6ba17a55290285d9ea8d269d0d6fb22721299d" exitCode=0 Oct 30 04:06:59.542264 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.542155 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" event={"ID":"69251e1c-69e6-49ed-ac90-6d8edb9f64ca","Type":"ContainerDied","Data":"a7cc62ac43e12846f507b8519a6ba17a55290285d9ea8d269d0d6fb22721299d"} Oct 30 04:06:59.542264 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.542208 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" event={"ID":"69251e1c-69e6-49ed-ac90-6d8edb9f64ca","Type":"ContainerDied","Data":"778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69"} Oct 30 04:06:59.542264 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.542225 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778fcbb1932c26fede743785aca0264f589beff9f986b6d53c2296478faa5e69" Oct 30 04:06:59.544306 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.544285 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 04:06:59.622827 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.622795 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\") pod \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " Oct 30 04:06:59.623009 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.622835 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\") pod \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " Oct 30 04:06:59.623009 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.622915 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\" (UniqueName: \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\") pod \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " Oct 30 04:06:59.623124 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.623025 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\") pod \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\" (UID: \"69251e1c-69e6-49ed-ac90-6d8edb9f64ca\") " Oct 30 04:06:59.623296 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.623271 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca" pod "69251e1c-69e6-49ed-ac90-6d8edb9f64ca" (UID: "69251e1c-69e6-49ed-ac90-6d8edb9f64ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:59.623708 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.623678 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.625190 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.625160 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls" pod "69251e1c-69e6-49ed-ac90-6d8edb9f64ca" (UID: "69251e1c-69e6-49ed-ac90-6d8edb9f64ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:59.625642 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.625619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh" pod "69251e1c-69e6-49ed-ac90-6d8edb9f64ca" (UID: "69251e1c-69e6-49ed-ac90-6d8edb9f64ca"). InnerVolumeSpecName "kube-api-access-sr2kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:59.625744 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.625684 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs" pod "69251e1c-69e6-49ed-ac90-6d8edb9f64ca" (UID: "69251e1c-69e6-49ed-ac90-6d8edb9f64ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:59.739109 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.739069 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.739109 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.739112 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr2kh\" (UniqueName: \"kubernetes.io/projected/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-kube-api-access-sr2kh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.739379 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:06:59.739127 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69251e1c-69e6-49ed-ac90-6d8edb9f64ca-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:00.551193 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:00.551156 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj" Oct 30 04:07:00.607723 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:00.607683 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 04:07:00.619067 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:00.619030 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10c0f-755d9b5cd-lz7lj"] Oct 30 04:07:02.155577 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:02.155543 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" path="/var/lib/kubelet/pods/69251e1c-69e6-49ed-ac90-6d8edb9f64ca/volumes" Oct 30 04:07:04.277600 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.277555 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" podUID="80f3c979-5311-4510-acef-256e2460002a" containerName="payload-processing" containerID="cri-o://d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663" gracePeriod=25 Oct 30 04:07:04.560506 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.560477 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 04:07:04.582409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.582379 4560 generic.go:356] "Generic (PLEG): container finished" podID="80f3c979-5311-4510-acef-256e2460002a" containerID="d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663" exitCode=0 Oct 30 04:07:04.582586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.582422 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" Oct 30 04:07:04.582586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.582428 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" event={"ID":"80f3c979-5311-4510-acef-256e2460002a","Type":"ContainerDied","Data":"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663"} Oct 30 04:07:04.582586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.582470 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf" event={"ID":"80f3c979-5311-4510-acef-256e2460002a","Type":"ContainerDied","Data":"4057a8c1ee97253cb2fbedda3dc9582cee1a784baa898a8ccc7bd75ea47d99c6"} Oct 30 04:07:04.582586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.582493 4560 scope.go:122] "RemoveContainer" containerID="d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663" Oct 30 04:07:04.592028 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.592011 4560 scope.go:122] "RemoveContainer" containerID="d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663" Oct 30 04:07:04.592294 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:04.592275 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663\": container with ID starting with d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663 not found: ID does not exist" containerID="d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663" Oct 30 04:07:04.592342 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.592300 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663"} err="failed to get container status \"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663\": rpc error: code = NotFound desc = could not find container \"d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663\": container with ID starting with d31e896e2fa7b92f01dd41bb5b8e5e61f349bbaaacdf5e8ceb0e0c0981de8663 not found: ID does not exist" Oct 30 04:07:04.695965 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.695934 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\" (UniqueName: \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\") pod \"80f3c979-5311-4510-acef-256e2460002a\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " Oct 30 04:07:04.696167 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.696104 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\") pod \"80f3c979-5311-4510-acef-256e2460002a\" (UID: \"80f3c979-5311-4510-acef-256e2460002a\") " Oct 30 04:07:04.696487 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.696462 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume" pod "80f3c979-5311-4510-acef-256e2460002a" (UID: "80f3c979-5311-4510-acef-256e2460002a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:04.698116 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.698094 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x" pod "80f3c979-5311-4510-acef-256e2460002a" (UID: "80f3c979-5311-4510-acef-256e2460002a"). InnerVolumeSpecName "kube-api-access-t6h9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:04.797570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.797467 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f3c979-5311-4510-acef-256e2460002a-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:04.797570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.797498 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6h9x\" (UniqueName: \"kubernetes.io/projected/80f3c979-5311-4510-acef-256e2460002a-kube-api-access-t6h9x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:04.929706 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.929663 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 04:07:04.932129 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:04.932096 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10c0f-64b64759f4-pchdf"] Oct 30 04:07:06.152472 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.152441 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f3c979-5311-4510-acef-256e2460002a" path="/var/lib/kubelet/pods/80f3c979-5311-4510-acef-256e2460002a/volumes" Oct 30 04:07:06.340205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.340166 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 04:07:06.340417 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.340357 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" podUID="e96b92c1-f79a-46e3-8ecf-74323753dd00" containerName="maas-api" containerID="cri-o://2563b0e013faf340af012ef41b5168f3c64de11542b52c050c46544b164113ef" gracePeriod=30 Oct 30 04:07:06.615456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.614200 4560 generic.go:356] "Generic (PLEG): container finished" podID="e96b92c1-f79a-46e3-8ecf-74323753dd00" containerID="2563b0e013faf340af012ef41b5168f3c64de11542b52c050c46544b164113ef" exitCode=0 Oct 30 04:07:06.615456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.614277 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" event={"ID":"e96b92c1-f79a-46e3-8ecf-74323753dd00","Type":"ContainerDied","Data":"2563b0e013faf340af012ef41b5168f3c64de11542b52c050c46544b164113ef"} Oct 30 04:07:06.615456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.614359 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" event={"ID":"e96b92c1-f79a-46e3-8ecf-74323753dd00","Type":"ContainerDied","Data":"906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842"} Oct 30 04:07:06.615456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.614384 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="906ec3a576dea18fe502fa8ffc2bf64b5f6f2c88dbdc81962499102285676842" Oct 30 04:07:06.627939 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.627908 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 04:07:06.723266 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.723170 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") pod \"e96b92c1-f79a-46e3-8ecf-74323753dd00\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " Oct 30 04:07:06.723443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.723308 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\" (UniqueName: \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\") pod \"e96b92c1-f79a-46e3-8ecf-74323753dd00\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " Oct 30 04:07:06.723443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.723380 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\") pod \"e96b92c1-f79a-46e3-8ecf-74323753dd00\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " Oct 30 04:07:06.723443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.723425 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\") pod \"e96b92c1-f79a-46e3-8ecf-74323753dd00\" (UID: \"e96b92c1-f79a-46e3-8ecf-74323753dd00\") " Oct 30 04:07:06.724125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.724088 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca" pod "e96b92c1-f79a-46e3-8ecf-74323753dd00" (UID: "e96b92c1-f79a-46e3-8ecf-74323753dd00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:06.724405 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.724382 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96b92c1-f79a-46e3-8ecf-74323753dd00-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:06.726210 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.726175 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls" pod "e96b92c1-f79a-46e3-8ecf-74323753dd00" (UID: "e96b92c1-f79a-46e3-8ecf-74323753dd00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:06.728072 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.727967 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs" pod "e96b92c1-f79a-46e3-8ecf-74323753dd00" (UID: "e96b92c1-f79a-46e3-8ecf-74323753dd00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:06.728191 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.728086 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558" pod "e96b92c1-f79a-46e3-8ecf-74323753dd00" (UID: "e96b92c1-f79a-46e3-8ecf-74323753dd00"). InnerVolumeSpecName "kube-api-access-c7558". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:06.825716 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.825674 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7558\" (UniqueName: \"kubernetes.io/projected/e96b92c1-f79a-46e3-8ecf-74323753dd00-kube-api-access-c7558\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:06.825716 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.825711 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:06.825716 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:06.825724 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96b92c1-f79a-46e3-8ecf-74323753dd00-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.234572 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.234529 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:07:07.235361 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235337 4560 cpu_manager.go:414] "containerMap: removing container" podUID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" containerName="maas-api" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235364 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" containerName="maas-api" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235392 4560 cpu_manager.go:414] "containerMap: removing container" podUID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" containerName="maas-api" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235402 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" containerName="maas-api" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235417 4560 cpu_manager.go:414] "containerMap: removing container" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235426 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" Oct 30 04:07:07.235457 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235455 4560 cpu_manager.go:414] "containerMap: removing container" podUID="80f3c979-5311-4510-acef-256e2460002a" containerName="payload-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235462 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f3c979-5311-4510-acef-256e2460002a" containerName="payload-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235485 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235493 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235506 4560 cpu_manager.go:414] "containerMap: removing container" podUID="087fc4d2-134e-44b9-aa65-7385b9f96b19" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235530 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087fc4d2-134e-44b9-aa65-7385b9f96b19" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235566 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7db61479-cb23-4486-9b05-f99634f64df7" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235574 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db61479-cb23-4486-9b05-f99634f64df7" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235586 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e96b92c1-f79a-46e3-8ecf-74323753dd00" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235593 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96b92c1-f79a-46e3-8ecf-74323753dd00" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235610 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235618 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235737 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f93420-c8bc-44f8-a5f3-d0d4ad165920" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235756 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="69251e1c-69e6-49ed-ac90-6d8edb9f64ca" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235767 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f7a842-978b-4921-a056-267eca8e8cab" containerName="payload-pre-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235781 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f3c979-5311-4510-acef-256e2460002a" containerName="payload-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235791 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb0bb28-a713-4d4b-b5c4-b882416ddea7" containerName="payload-pre-processing" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235802 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="087fc4d2-134e-44b9-aa65-7385b9f96b19" containerName="maas-api" Oct 30 04:07:07.235811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235815 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db61479-cb23-4486-9b05-f99634f64df7" containerName="maas-api" Oct 30 04:07:07.236951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235824 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96b92c1-f79a-46e3-8ecf-74323753dd00" containerName="maas-api" Oct 30 04:07:07.236951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.235834 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47f8638-835e-422b-bb51-2dc58acba6e1" containerName="payload-processing" Oct 30 04:07:07.239938 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.239904 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.256862 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.256832 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-30e3dd\"/\"openshift-service-ca.crt\"" Oct 30 04:07:07.256997 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.256868 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-30e3dd\"/\"default-dockercfg-4s7nb\"" Oct 30 04:07:07.256997 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.256913 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-30e3dd\"/\"kube-root-ca.crt\"" Oct 30 04:07:07.256997 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.256937 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-30e3dd\"/\"rate-test-model-30e3dd-kserve-self-signed-certs\"" Oct 30 04:07:07.257952 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.257932 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:07:07.332104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332054 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4n8\" (UniqueName: \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.332289 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332244 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.332351 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332299 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.332454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332432 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.332559 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332510 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.332669 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.332645 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434179 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434140 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434196 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434222 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434272 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434305 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434367 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4n8\" (UniqueName: \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434646 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.434761 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.434734 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.435114 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.435087 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.436807 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.436784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.437025 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.437003 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.454456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.454421 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4n8\" (UniqueName: \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\") pod \"rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.557843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.557754 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:07.622437 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.622413 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc" Oct 30 04:07:07.681068 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.681025 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 04:07:07.686691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.686663 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-817531-576bc4cf5f-j9xbc"] Oct 30 04:07:07.884344 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:07.884315 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:07:07.885002 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:07:07.884973 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc937282b_846b_455f_8be1_f63d060653ac.slice/crio-ad6ea070834628f4fd6dd463291ba741143ddc27899fef253a803e7bc9f3039a WatchSource:0}: Error finding container ad6ea070834628f4fd6dd463291ba741143ddc27899fef253a803e7bc9f3039a: Status 404 returned error can't find the container with id ad6ea070834628f4fd6dd463291ba741143ddc27899fef253a803e7bc9f3039a Oct 30 04:07:08.156115 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:08.156033 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96b92c1-f79a-46e3-8ecf-74323753dd00" path="/var/lib/kubelet/pods/e96b92c1-f79a-46e3-8ecf-74323753dd00/volumes" Oct 30 04:07:08.629562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:08.629494 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" event={"ID":"c937282b-846b-455f-8be1-f63d060653ac","Type":"ContainerStarted","Data":"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887"} Oct 30 04:07:08.629562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:08.629565 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" event={"ID":"c937282b-846b-455f-8be1-f63d060653ac","Type":"ContainerStarted","Data":"ad6ea070834628f4fd6dd463291ba741143ddc27899fef253a803e7bc9f3039a"} Oct 30 04:07:08.671878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:08.671823 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" podStartSLOduration=1.6718083620000002 podStartE2EDuration="1.671808362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:08.670814872 +0000 UTC m=+3856.655934762" watchObservedRunningTime="2026-10-30 04:07:08.671808362 +0000 UTC m=+3856.656928249" Oct 30 04:07:09.558249 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:09.558212 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:11.749906 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:11.749858 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:07:17.559018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:17.558976 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:17.584225 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:17.584185 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:17.693495 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:17.693462 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:07:18.744668 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:18.744626 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 04:07:18.757883 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:18.757848 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 04:07:22.335239 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.335205 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:07:22.340680 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.340651 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.358167 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.358140 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:07:22.367916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.367864 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-dd81fd\"/\"default-dockercfg-29268\"" Oct 30 04:07:22.368073 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.367964 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-dd81fd\"/\"rate-test-model-dd81fd-kserve-self-signed-certs\"" Oct 30 04:07:22.432782 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.432743 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.432979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.432821 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.433033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.432995 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.433080 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.433067 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.433155 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.433134 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8f4z\" (UniqueName: \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.433207 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.433179 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.436960 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.436937 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-dd81fd\"/\"openshift-service-ca.crt\"" Oct 30 04:07:22.436960 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.436943 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-dd81fd\"/\"kube-root-ca.crt\"" Oct 30 04:07:22.535018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.534974 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535189 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535069 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535189 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535170 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535297 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535224 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535297 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8f4z\" (UniqueName: \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535481 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535444 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535610 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535456 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535676 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535601 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.535676 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.535657 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.537815 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.537779 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.537981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.537962 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.554850 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.554822 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8f4z\" (UniqueName: \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\") pod \"rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.656642 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.656542 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:22.964743 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:22.964710 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:07:22.966295 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:07:22.966269 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cc218d5_1147_4e69_a7a1_618b7b33e489.slice/crio-f075acc3b66f38313a7e8d730d729bf853638eafbb781b2952effc0686a074ff WatchSource:0}: Error finding container f075acc3b66f38313a7e8d730d729bf853638eafbb781b2952effc0686a074ff: Status 404 returned error can't find the container with id f075acc3b66f38313a7e8d730d729bf853638eafbb781b2952effc0686a074ff Oct 30 04:07:23.734704 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:23.734667 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" event={"ID":"7cc218d5-1147-4e69-a7a1-618b7b33e489","Type":"ContainerStarted","Data":"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c"} Oct 30 04:07:23.735163 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:23.734714 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" event={"ID":"7cc218d5-1147-4e69-a7a1-618b7b33e489","Type":"ContainerStarted","Data":"f075acc3b66f38313a7e8d730d729bf853638eafbb781b2952effc0686a074ff"} Oct 30 04:07:23.785837 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:23.785780 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" podStartSLOduration=1.785765548 podStartE2EDuration="1.785765548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:23.784549589 +0000 UTC m=+3871.769669490" watchObservedRunningTime="2026-10-30 04:07:23.785765548 +0000 UTC m=+3871.770885436" Oct 30 04:07:24.673541 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:24.657639 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:29.116493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:29.116451 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 04:07:29.117047 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:29.116728 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" containerID="cri-o://16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd" gracePeriod=30 Oct 30 04:07:32.657607 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:32.657569 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:32.671060 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:32.671030 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:32.801126 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:32.801092 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:07:33.767678 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:33.767632 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" podUID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" containerName="main" containerID="cri-o://c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1" gracePeriod=45 Oct 30 04:07:33.778992 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:33.778954 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" podUID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" containerName="main" containerID="cri-o://0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e" gracePeriod=45 Oct 30 04:07:34.049269 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.049235 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 04:07:34.077843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.077817 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 04:07:34.088241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088171 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.088795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088482 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\" (UniqueName: \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.088795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088590 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.088795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088644 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\" (UniqueName: \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.088795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088664 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.088795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.088690 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\" (UniqueName: \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.089691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.089633 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.090302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.090270 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.091775 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.091748 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "kube-api-access-45mhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:34.092397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.092371 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:34.092873 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.092851 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "kube-api-access-hfm94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:34.093647 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.093623 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.189760 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.189726 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.189958 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.189779 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.190062 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190035 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.190142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190054 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.190215 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190201 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.190426 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190401 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.190571 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190506 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\") pod \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\" (UID: \"e556d36d-074d-4f34-b96d-5c6b9b35dd77\") " Oct 30 04:07:34.190571 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190561 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\") pod \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\" (UID: \"4e9d1e24-9e81-47d0-81fc-85f9662c41bc\") " Oct 30 04:07:34.190732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190712 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home" pod "e556d36d-074d-4f34-b96d-5c6b9b35dd77" (UID: "e556d36d-074d-4f34-b96d-5c6b9b35dd77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.190850 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.190810 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.191312 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191290 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191748 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191728 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191752 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191768 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191785 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191799 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e556d36d-074d-4f34-b96d-5c6b9b35dd77-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191812 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191825 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfm94\" (UniqueName: \"kubernetes.io/projected/e556d36d-074d-4f34-b96d-5c6b9b35dd77-kube-api-access-hfm94\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.191842 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191838 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e556d36d-074d-4f34-b96d-5c6b9b35dd77-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.192127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.191852 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mhr\" (UniqueName: \"kubernetes.io/projected/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-kube-api-access-45mhr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.192127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.192055 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:34.192127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.192052 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm" pod "4e9d1e24-9e81-47d0-81fc-85f9662c41bc" (UID: "4e9d1e24-9e81-47d0-81fc-85f9662c41bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.292969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.292886 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.292969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.292915 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1e24-9e81-47d0-81fc-85f9662c41bc-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.378130 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.378099 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 04:07:34.393394 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393359 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393432 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393504 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393583 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393803 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393649 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393803 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393757 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.393803 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393765 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393974 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393857 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.393974 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393882 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:34.393974 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393961 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.394149 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.393996 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\") pod \"61014647-b515-45f6-886c-87b102274c29\" (UID: \"61014647-b515-45f6-886c-87b102274c29\") " Oct 30 04:07:34.394149 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394132 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.394336 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394315 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.394402 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394366 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.394529 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394495 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.394593 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394536 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.394593 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394547 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.394593 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394555 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61014647-b515-45f6-886c-87b102274c29-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.394593 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.394563 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.396096 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.396077 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "kube-api-access-dvzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:34.396172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.396087 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:34.396300 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.396286 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:07:34.396340 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.396302 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy" pod "61014647-b515-45f6-886c-87b102274c29" (UID: "61014647-b515-45f6-886c-87b102274c29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.495381 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.495345 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61014647-b515-45f6-886c-87b102274c29-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.495381 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.495376 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzxh\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-kube-api-access-dvzxh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.495381 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.495387 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61014647-b515-45f6-886c-87b102274c29-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.495674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.495396 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61014647-b515-45f6-886c-87b102274c29-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.802145 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.802115 4560 generic.go:356] "Generic (PLEG): container finished" podID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" containerID="0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e" exitCode=0 Oct 30 04:07:34.802588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.802157 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" Oct 30 04:07:34.802588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.802197 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" event={"ID":"4e9d1e24-9e81-47d0-81fc-85f9662c41bc","Type":"ContainerDied","Data":"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e"} Oct 30 04:07:34.802588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.802239 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5" event={"ID":"4e9d1e24-9e81-47d0-81fc-85f9662c41bc","Type":"ContainerDied","Data":"e5300195045ce111a193acd94cb2dbbf0eadf2864fcdead32c05843553c803a8"} Oct 30 04:07:34.802588 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.802256 4560 scope.go:122] "RemoveContainer" containerID="0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e" Oct 30 04:07:34.803700 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.803683 4560 generic.go:356] "Generic (PLEG): container finished" podID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" containerID="c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1" exitCode=0 Oct 30 04:07:34.803800 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.803783 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" event={"ID":"e556d36d-074d-4f34-b96d-5c6b9b35dd77","Type":"ContainerDied","Data":"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1"} Oct 30 04:07:34.803853 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.803816 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" event={"ID":"e556d36d-074d-4f34-b96d-5c6b9b35dd77","Type":"ContainerDied","Data":"81a981d8dfbc85d9d5d8d31681b80ee5da20cd076b17e6f74559029020860825"} Oct 30 04:07:34.803897 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.803874 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5" Oct 30 04:07:34.811042 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.811020 4560 generic.go:356] "Generic (PLEG): container finished" podID="61014647-b515-45f6-886c-87b102274c29" containerID="16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd" exitCode=0 Oct 30 04:07:34.811178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.811066 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" Oct 30 04:07:34.811178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.811104 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" event={"ID":"61014647-b515-45f6-886c-87b102274c29","Type":"ContainerDied","Data":"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd"} Oct 30 04:07:34.811178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.811157 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7" event={"ID":"61014647-b515-45f6-886c-87b102274c29","Type":"ContainerDied","Data":"95c360b784a66c1caff650ee573d66974b1cea6fccec13c564d0e682595d7e9f"} Oct 30 04:07:34.813823 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.813684 4560 scope.go:122] "RemoveContainer" containerID="0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e" Oct 30 04:07:34.814056 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:34.814032 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e\": container with ID starting with 0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e not found: ID does not exist" containerID="0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e" Oct 30 04:07:34.814162 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.814062 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e"} err="failed to get container status \"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e\": rpc error: code = NotFound desc = could not find container \"0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e\": container with ID starting with 0b4a607f6c8d02035db5c250973da3313c2aa435cf1b49d3383affbd28b0c81e not found: ID does not exist" Oct 30 04:07:34.814162 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.814084 4560 scope.go:122] "RemoveContainer" containerID="c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1" Oct 30 04:07:34.823296 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.823279 4560 scope.go:122] "RemoveContainer" containerID="c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1" Oct 30 04:07:34.823616 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:34.823591 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1\": container with ID starting with c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1 not found: ID does not exist" containerID="c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1" Oct 30 04:07:34.823723 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.823621 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1"} err="failed to get container status \"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1\": rpc error: code = NotFound desc = could not find container \"c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1\": container with ID starting with c66b7e32b4749e95be56b2608c5eda352c3b72cc9c4f16760f622338799710e1 not found: ID does not exist" Oct 30 04:07:34.823723 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.823636 4560 scope.go:122] "RemoveContainer" containerID="16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd" Oct 30 04:07:34.832329 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.832304 4560 scope.go:122] "RemoveContainer" containerID="16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd" Oct 30 04:07:34.832617 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:34.832598 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd\": container with ID starting with 16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd not found: ID does not exist" containerID="16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd" Oct 30 04:07:34.832683 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.832626 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd"} err="failed to get container status \"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd\": rpc error: code = NotFound desc = could not find container \"16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd\": container with ID starting with 16430ed9b0f92456dff75a2fc84ba93c69f8fee224cc24382f6b2315fbb027fd not found: ID does not exist" Oct 30 04:07:34.868649 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.868622 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 04:07:34.870853 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.870831 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/simulator-w0-a9c9b6-kserve-58675477f7-8ndd5"] Oct 30 04:07:34.904332 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.904297 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 04:07:34.906652 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.906628 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/premium-w0-a9c9b6-kserve-5cf68468d8-r58n5"] Oct 30 04:07:34.939885 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.939835 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 04:07:34.942642 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:34.942614 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-xzlj7"] Oct 30 04:07:36.153407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:36.153374 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" path="/var/lib/kubelet/pods/4e9d1e24-9e81-47d0-81fc-85f9662c41bc/volumes" Oct 30 04:07:36.153925 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:36.153909 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61014647-b515-45f6-886c-87b102274c29" path="/var/lib/kubelet/pods/61014647-b515-45f6-886c-87b102274c29/volumes" Oct 30 04:07:36.154317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:36.154304 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" path="/var/lib/kubelet/pods/e556d36d-074d-4f34-b96d-5c6b9b35dd77/volumes" Oct 30 04:07:41.763578 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:07:41.763541 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:07:50.617649 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.617610 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 04:07:50.618551 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.618494 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" podUID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" containerName="maas-api" containerID="cri-o://5da0dd9efab6d9517fa816e4a7517355bf28fa8e7ddd666905e539db351aeebc" gracePeriod=30 Oct 30 04:07:50.918118 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.918085 4560 generic.go:356] "Generic (PLEG): container finished" podID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" containerID="5da0dd9efab6d9517fa816e4a7517355bf28fa8e7ddd666905e539db351aeebc" exitCode=0 Oct 30 04:07:50.918282 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.918160 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" event={"ID":"f3f8c88a-6b87-4e53-8b33-ed348f369c7b","Type":"ContainerDied","Data":"5da0dd9efab6d9517fa816e4a7517355bf28fa8e7ddd666905e539db351aeebc"} Oct 30 04:07:50.918282 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.918263 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" event={"ID":"f3f8c88a-6b87-4e53-8b33-ed348f369c7b","Type":"ContainerDied","Data":"a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7"} Oct 30 04:07:50.918371 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.918288 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a601e069c8ef582f2241b73cd65afb9d1bb076d9449284e236dbea9c34f07be7" Oct 30 04:07:50.949350 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:50.949316 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 04:07:51.109131 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.109026 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\") pod \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " Oct 30 04:07:51.109131 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.109106 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\") pod \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " Oct 30 04:07:51.109409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.109167 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\") pod \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " Oct 30 04:07:51.109409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.109225 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\" (UniqueName: \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\") pod \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\" (UID: \"f3f8c88a-6b87-4e53-8b33-ed348f369c7b\") " Oct 30 04:07:51.110012 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.109977 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca" pod "f3f8c88a-6b87-4e53-8b33-ed348f369c7b" (UID: "f3f8c88a-6b87-4e53-8b33-ed348f369c7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:51.111935 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.111903 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn" pod "f3f8c88a-6b87-4e53-8b33-ed348f369c7b" (UID: "f3f8c88a-6b87-4e53-8b33-ed348f369c7b"). InnerVolumeSpecName "kube-api-access-qhngn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:51.111935 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.111917 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs" pod "f3f8c88a-6b87-4e53-8b33-ed348f369c7b" (UID: "f3f8c88a-6b87-4e53-8b33-ed348f369c7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:51.112080 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.111934 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls" pod "f3f8c88a-6b87-4e53-8b33-ed348f369c7b" (UID: "f3f8c88a-6b87-4e53-8b33-ed348f369c7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:51.210575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.210544 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:51.210575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.210575 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhngn\" (UniqueName: \"kubernetes.io/projected/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-kube-api-access-qhngn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:51.210790 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.210589 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:51.210790 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.210601 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f8c88a-6b87-4e53-8b33-ed348f369c7b-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:07:51.923127 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.923094 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk" Oct 30 04:07:51.976220 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.976184 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 04:07:51.989027 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:51.988996 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7d99a4-78d45c58b5-rwlmk"] Oct 30 04:07:52.152743 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:52.152707 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" path="/var/lib/kubelet/pods/f3f8c88a-6b87-4e53-8b33-ed348f369c7b/volumes" Oct 30 04:07:53.687689 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:53.687654 4560 scope.go:122] "RemoveContainer" containerID="2563b0e013faf340af012ef41b5168f3c64de11542b52c050c46544b164113ef" Oct 30 04:07:53.697231 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:53.697203 4560 scope.go:122] "RemoveContainer" containerID="5da0dd9efab6d9517fa816e4a7517355bf28fa8e7ddd666905e539db351aeebc" Oct 30 04:07:53.706382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:53.706360 4560 scope.go:122] "RemoveContainer" containerID="a7cc62ac43e12846f507b8519a6ba17a55290285d9ea8d269d0d6fb22721299d" Oct 30 04:07:55.931358 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:55.931317 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:07:55.931908 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:07:55.931588 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" containerID="cri-o://1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b" gracePeriod=30 Oct 30 04:08:01.214067 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.214041 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:08:01.319789 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.319753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.319940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.319846 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.319940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.319875 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.319940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.319903 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320112 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320073 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.320168 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320084 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320211 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320159 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.320211 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320177 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320211 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320195 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.320362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320211 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320244 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320304 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\") pod \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\" (UID: \"7e6ccbee-83a6-4bad-a865-21018fa6e9fa\") " Oct 30 04:08:01.320569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320547 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:01.320674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.320649 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.321169 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.321141 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.321169 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.321169 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.321352 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.321184 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.321352 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.321200 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.321352 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.321214 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.322461 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.322432 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:01.322695 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.322672 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.322802 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.322701 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "kube-api-access-kdbwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:01.322900 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.322885 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo" pod "7e6ccbee-83a6-4bad-a865-21018fa6e9fa" (UID: "7e6ccbee-83a6-4bad-a865-21018fa6e9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:01.422910 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.422877 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.422910 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.422907 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.422910 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.422915 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.422910 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.422924 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbwq\" (UniqueName: \"kubernetes.io/projected/7e6ccbee-83a6-4bad-a865-21018fa6e9fa-kube-api-access-kdbwq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.992818 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.992784 4560 generic.go:356] "Generic (PLEG): container finished" podID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerID="1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b" exitCode=0 Oct 30 04:08:01.993011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.992867 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" Oct 30 04:08:01.993011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.992865 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" event={"ID":"7e6ccbee-83a6-4bad-a865-21018fa6e9fa","Type":"ContainerDied","Data":"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b"} Oct 30 04:08:01.993147 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.993028 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw" event={"ID":"7e6ccbee-83a6-4bad-a865-21018fa6e9fa","Type":"ContainerDied","Data":"cff8e02d9e402402787d46ed21d84717a7f62ea9b84c6bba88fac3469cbe8cac"} Oct 30 04:08:01.993147 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:01.993063 4560 scope.go:122] "RemoveContainer" containerID="1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b" Oct 30 04:08:02.003190 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:02.003140 4560 scope.go:122] "RemoveContainer" containerID="1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b" Oct 30 04:08:02.003434 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:08:02.003403 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b\": container with ID starting with 1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b not found: ID does not exist" containerID="1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b" Oct 30 04:08:02.003625 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:02.003444 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b"} err="failed to get container status \"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b\": rpc error: code = NotFound desc = could not find container \"1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b\": container with ID starting with 1c3803cf5e14400a4361c5f76967c747f9a234d8316c0d875fb05dcc3dcc010b not found: ID does not exist" Oct 30 04:08:02.058939 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:02.058895 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:08:02.071777 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:02.071739 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-7rlrw"] Oct 30 04:08:02.152216 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:02.152181 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" path="/var/lib/kubelet/pods/7e6ccbee-83a6-4bad-a865-21018fa6e9fa/volumes" Oct 30 04:08:11.750162 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:08:11.750075 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:08:18.957944 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:18.957908 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 04:08:18.988371 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:18.988337 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 04:08:33.979061 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:33.979020 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerName="main" containerID="cri-o://66fa9abf1fffe51da51235f39e80d57411e9e7dc9a21da367689da708716554f" gracePeriod=45 Oct 30 04:08:34.009510 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.009472 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerName="main" containerID="cri-o://b901c16564919170acfc54051adb0913066ef802acbf42ef21ee1818b1551ddc" gracePeriod=45 Oct 30 04:08:34.019043 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.019010 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused" Oct 30 04:08:34.169793 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.169754 4560 generic.go:356] "Generic (PLEG): container finished" podID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerID="66fa9abf1fffe51da51235f39e80d57411e9e7dc9a21da367689da708716554f" exitCode=0 Oct 30 04:08:34.169966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.169848 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" event={"ID":"d833a78d-8e47-45ea-9e16-0b503352a97c","Type":"ContainerDied","Data":"66fa9abf1fffe51da51235f39e80d57411e9e7dc9a21da367689da708716554f"} Oct 30 04:08:34.171315 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.171291 4560 generic.go:356] "Generic (PLEG): container finished" podID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerID="b901c16564919170acfc54051adb0913066ef802acbf42ef21ee1818b1551ddc" exitCode=0 Oct 30 04:08:34.171440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.171348 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" event={"ID":"eee2fe47-8e05-4edc-b0c2-b15bdac78880","Type":"ContainerDied","Data":"b901c16564919170acfc54051adb0913066ef802acbf42ef21ee1818b1551ddc"} Oct 30 04:08:34.305118 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.305088 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 04:08:34.324171 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.324096 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 04:08:34.393840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.393803 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.393973 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394024 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394060 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394168 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\" (UniqueName: \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394241 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394269 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\" (UniqueName: \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394316 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394346 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394354 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.394428 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394424 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\") pod \"d833a78d-8e47-45ea-9e16-0b503352a97c\" (UID: \"d833a78d-8e47-45ea-9e16-0b503352a97c\") " Oct 30 04:08:34.395044 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394492 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\" (UniqueName: \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.395044 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.394564 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\") pod \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\" (UID: \"eee2fe47-8e05-4edc-b0c2-b15bdac78880\") " Oct 30 04:08:34.395044 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.395029 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.398565 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.395656 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.398565 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.396101 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.398565 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.396816 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.398565 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.397252 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.398565 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.397445 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.399101 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.398585 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.399101 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.399070 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:34.399655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.399626 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "kube-api-access-s6ws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:34.401277 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.401249 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "kube-api-access-pl89s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:34.401802 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.401778 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs" pod "eee2fe47-8e05-4edc-b0c2-b15bdac78880" (UID: "eee2fe47-8e05-4edc-b0c2-b15bdac78880"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:34.401802 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.401790 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm" pod "d833a78d-8e47-45ea-9e16-0b503352a97c" (UID: "d833a78d-8e47-45ea-9e16-0b503352a97c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.496408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496369 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496401 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496411 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d833a78d-8e47-45ea-9e16-0b503352a97c-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496437 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496446 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496455 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ws8\" (UniqueName: \"kubernetes.io/projected/d833a78d-8e47-45ea-9e16-0b503352a97c-kube-api-access-s6ws8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496464 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496474 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl89s\" (UniqueName: \"kubernetes.io/projected/eee2fe47-8e05-4edc-b0c2-b15bdac78880-kube-api-access-pl89s\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496481 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496489 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee2fe47-8e05-4edc-b0c2-b15bdac78880-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.496685 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:34.496496 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d833a78d-8e47-45ea-9e16-0b503352a97c-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:08:35.178071 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.178039 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" Oct 30 04:08:35.178568 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.178039 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk" event={"ID":"d833a78d-8e47-45ea-9e16-0b503352a97c","Type":"ContainerDied","Data":"6d2be694dea87242818096c3c35faf8ae041f257497437ab8283451c1da7aa69"} Oct 30 04:08:35.178568 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.178172 4560 scope.go:122] "RemoveContainer" containerID="66fa9abf1fffe51da51235f39e80d57411e9e7dc9a21da367689da708716554f" Oct 30 04:08:35.179684 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.179609 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" event={"ID":"eee2fe47-8e05-4edc-b0c2-b15bdac78880","Type":"ContainerDied","Data":"3aab08d2eff59e8752146becb2f3ee4bfca32843237c31c1f734942d3a05c26a"} Oct 30 04:08:35.179684 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.179641 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" Oct 30 04:08:35.198274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.197766 4560 scope.go:122] "RemoveContainer" containerID="b901c16564919170acfc54051adb0913066ef802acbf42ef21ee1818b1551ddc" Oct 30 04:08:35.231362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.231328 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 04:08:35.250627 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.246025 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/distinct-w4-8d80b1-kserve-9fb458976-grpsk"] Oct 30 04:08:35.250627 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.249712 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:08:35.291598 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.284581 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 04:08:35.291598 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:35.287856 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/simulator-w4-8d80b1-kserve-576844c4bd-cdsl6"] Oct 30 04:08:36.160241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:36.160163 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" path="/var/lib/kubelet/pods/d833a78d-8e47-45ea-9e16-0b503352a97c/volumes" Oct 30 04:08:36.160966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:08:36.160939 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" path="/var/lib/kubelet/pods/eee2fe47-8e05-4edc-b0c2-b15bdac78880/volumes" Oct 30 04:08:41.751203 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:08:41.751156 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:09:11.749565 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:09:11.749506 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:09:27.758194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:27.758156 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:09:33.096702 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:33.096667 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:09:41.750406 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:09:41.750325 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:09:42.778999 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:42.778946 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" podUID="c937282b-846b-455f-8be1-f63d060653ac" containerName="main" containerID="cri-o://d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887" gracePeriod=45 Oct 30 04:09:43.028959 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.028931 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:09:43.169194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169095 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169159 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169203 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169248 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169271 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\" (UniqueName: \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169339 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:43.169456 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169343 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\") pod \"c937282b-846b-455f-8be1-f63d060653ac\" (UID: \"c937282b-846b-455f-8be1-f63d060653ac\") " Oct 30 04:09:43.169695 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169457 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:43.169695 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169557 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:43.169816 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169777 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.169816 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169794 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.169925 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.169821 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.171611 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.171584 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:43.171739 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.171648 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:09:43.171739 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.171694 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8" pod "c937282b-846b-455f-8be1-f63d060653ac" (UID: "c937282b-846b-455f-8be1-f63d060653ac"). InnerVolumeSpecName "kube-api-access-nw4n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:09:43.270439 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.270403 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c937282b-846b-455f-8be1-f63d060653ac-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.270439 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.270435 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4n8\" (UniqueName: \"kubernetes.io/projected/c937282b-846b-455f-8be1-f63d060653ac-kube-api-access-nw4n8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.270439 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.270445 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c937282b-846b-455f-8be1-f63d060653ac-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:43.545755 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.545722 4560 generic.go:356] "Generic (PLEG): container finished" podID="c937282b-846b-455f-8be1-f63d060653ac" containerID="d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887" exitCode=0 Oct 30 04:09:43.545976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.545770 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" Oct 30 04:09:43.545976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.545810 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" event={"ID":"c937282b-846b-455f-8be1-f63d060653ac","Type":"ContainerDied","Data":"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887"} Oct 30 04:09:43.545976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.545861 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj" event={"ID":"c937282b-846b-455f-8be1-f63d060653ac","Type":"ContainerDied","Data":"ad6ea070834628f4fd6dd463291ba741143ddc27899fef253a803e7bc9f3039a"} Oct 30 04:09:43.545976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.545883 4560 scope.go:122] "RemoveContainer" containerID="d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887" Oct 30 04:09:43.556867 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.556849 4560 scope.go:122] "RemoveContainer" containerID="d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887" Oct 30 04:09:43.557158 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:09:43.557132 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887\": container with ID starting with d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887 not found: ID does not exist" containerID="d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887" Oct 30 04:09:43.557215 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.557172 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887"} err="failed to get container status \"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887\": rpc error: code = NotFound desc = could not find container \"d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887\": container with ID starting with d7911d0f105a7f8952687251f2c3b4091ebe4a19cc57ff37b39bb8855ac8c887 not found: ID does not exist" Oct 30 04:09:43.597696 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.597663 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:09:43.599898 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:43.599871 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-30e3dd/rate-test-model-30e3dd-kserve-79b8f4f88f-w6pfj"] Oct 30 04:09:44.152302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:44.152267 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c937282b-846b-455f-8be1-f63d060653ac" path="/var/lib/kubelet/pods/c937282b-846b-455f-8be1-f63d060653ac/volumes" Oct 30 04:09:47.874888 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:47.874855 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:09:47.875411 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:47.875118 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" podUID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" containerName="maas-api" containerID="cri-o://e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b" gracePeriod=30 Oct 30 04:09:48.117856 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.117814 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" podUID="7cc218d5-1147-4e69-a7a1-618b7b33e489" containerName="main" containerID="cri-o://c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c" gracePeriod=45 Oct 30 04:09:48.402931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.402906 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:09:48.406099 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.406079 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:09:48.422101 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422075 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\") pod \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " Oct 30 04:09:48.422217 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422111 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\") pod \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " Oct 30 04:09:48.422217 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422132 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.422217 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422150 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.422383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422214 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\" (UniqueName: \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.422443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422396 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.422443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422410 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:48.422570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422458 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\") pod \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " Oct 30 04:09:48.422570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422505 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.422570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422561 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\" (UniqueName: \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\") pod \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\" (UID: \"b5cff3e2-50cd-4110-bf7a-2bebeb18f822\") " Oct 30 04:09:48.422741 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422574 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca" pod "b5cff3e2-50cd-4110-bf7a-2bebeb18f822" (UID: "b5cff3e2-50cd-4110-bf7a-2bebeb18f822"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:09:48.422741 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422696 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:48.422971 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.422948 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:48.423589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.423071 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\") pod \"7cc218d5-1147-4e69-a7a1-618b7b33e489\" (UID: \"7cc218d5-1147-4e69-a7a1-618b7b33e489\") " Oct 30 04:09:48.423589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.423491 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.423589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.423510 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.423589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.423555 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.423589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.423568 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.424976 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.424950 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls" pod "b5cff3e2-50cd-4110-bf7a-2bebeb18f822" (UID: "b5cff3e2-50cd-4110-bf7a-2bebeb18f822"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:09:48.425100 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.425077 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs" pod "b5cff3e2-50cd-4110-bf7a-2bebeb18f822" (UID: "b5cff3e2-50cd-4110-bf7a-2bebeb18f822"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:09:48.425204 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.425116 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:48.425305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.425286 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "kube-api-access-h8f4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:09:48.425444 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.425423 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw" pod "b5cff3e2-50cd-4110-bf7a-2bebeb18f822" (UID: "b5cff3e2-50cd-4110-bf7a-2bebeb18f822"). InnerVolumeSpecName "kube-api-access-qfvbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:09:48.425534 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.425446 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs" pod "7cc218d5-1147-4e69-a7a1-618b7b33e489" (UID: "7cc218d5-1147-4e69-a7a1-618b7b33e489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:09:48.524801 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524759 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.524801 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524796 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvbw\" (UniqueName: \"kubernetes.io/projected/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-kube-api-access-qfvbw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.525018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524811 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc218d5-1147-4e69-a7a1-618b7b33e489-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.525018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524825 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5cff3e2-50cd-4110-bf7a-2bebeb18f822-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.525018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524835 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc218d5-1147-4e69-a7a1-618b7b33e489-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.525018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.524846 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8f4z\" (UniqueName: \"kubernetes.io/projected/7cc218d5-1147-4e69-a7a1-618b7b33e489-kube-api-access-h8f4z\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:09:48.574038 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.574000 4560 generic.go:356] "Generic (PLEG): container finished" podID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" containerID="e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b" exitCode=0 Oct 30 04:09:48.574246 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.574055 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" Oct 30 04:09:48.574246 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.574090 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" event={"ID":"b5cff3e2-50cd-4110-bf7a-2bebeb18f822","Type":"ContainerDied","Data":"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b"} Oct 30 04:09:48.574246 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.574153 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq" event={"ID":"b5cff3e2-50cd-4110-bf7a-2bebeb18f822","Type":"ContainerDied","Data":"32c2ee7cd9db2c0f787fdf81dd5b2b29d0eea4ac6eed8f62a4da61fd73acbb8c"} Oct 30 04:09:48.574246 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.574186 4560 scope.go:122] "RemoveContainer" containerID="e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b" Oct 30 04:09:48.575460 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.575438 4560 generic.go:356] "Generic (PLEG): container finished" podID="7cc218d5-1147-4e69-a7a1-618b7b33e489" containerID="c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c" exitCode=0 Oct 30 04:09:48.575617 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.575488 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" Oct 30 04:09:48.575617 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.575545 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" event={"ID":"7cc218d5-1147-4e69-a7a1-618b7b33e489","Type":"ContainerDied","Data":"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c"} Oct 30 04:09:48.575617 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.575584 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf" event={"ID":"7cc218d5-1147-4e69-a7a1-618b7b33e489","Type":"ContainerDied","Data":"f075acc3b66f38313a7e8d730d729bf853638eafbb781b2952effc0686a074ff"} Oct 30 04:09:48.585597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.585579 4560 scope.go:122] "RemoveContainer" containerID="e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b" Oct 30 04:09:48.585911 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:09:48.585892 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b\": container with ID starting with e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b not found: ID does not exist" containerID="e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b" Oct 30 04:09:48.585987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.585922 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b"} err="failed to get container status \"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b\": rpc error: code = NotFound desc = could not find container \"e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b\": container with ID starting with e37d298889fde3e031555475e6a922919c3fa914890f379c34f6bd2fd032949b not found: ID does not exist" Oct 30 04:09:48.585987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.585946 4560 scope.go:122] "RemoveContainer" containerID="c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c" Oct 30 04:09:48.594382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.594356 4560 scope.go:122] "RemoveContainer" containerID="c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c" Oct 30 04:09:48.594685 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:09:48.594661 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c\": container with ID starting with c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c not found: ID does not exist" containerID="c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c" Oct 30 04:09:48.594789 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.594693 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c"} err="failed to get container status \"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c\": rpc error: code = NotFound desc = could not find container \"c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c\": container with ID starting with c4ab0f189c853e0105e1310c26061e074f54cd2414191a219928b5db6bafc14c not found: ID does not exist" Oct 30 04:09:48.628797 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.628764 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:09:48.634760 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.634689 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-dd81fd/rate-test-model-dd81fd-kserve-7975bcbcc-hfxrf"] Oct 30 04:09:48.670575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.670541 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:09:48.675277 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:48.675239 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30e3dd-6457c4c48d-wcxfq"] Oct 30 04:09:50.151914 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:50.151878 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc218d5-1147-4e69-a7a1-618b7b33e489" path="/var/lib/kubelet/pods/7cc218d5-1147-4e69-a7a1-618b7b33e489/volumes" Oct 30 04:09:50.152328 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:50.152314 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" path="/var/lib/kubelet/pods/b5cff3e2-50cd-4110-bf7a-2bebeb18f822/volumes" Oct 30 04:09:59.410199 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:59.410165 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:09:59.444604 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:09:59.444566 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:10:03.257843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.257796 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:10:03.258296 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.258038 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" podUID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" containerName="maas-api" containerID="cri-o://ca5de28dbacf4de5d751779cbd52c8cd3b5de9d0166df83166efaaef5db49ff3" gracePeriod=30 Oct 30 04:10:03.661177 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.661148 4560 generic.go:356] "Generic (PLEG): container finished" podID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" containerID="ca5de28dbacf4de5d751779cbd52c8cd3b5de9d0166df83166efaaef5db49ff3" exitCode=0 Oct 30 04:10:03.661348 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.661223 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" event={"ID":"85dff755-f0bf-49b5-b9bd-5c6fc649154f","Type":"ContainerDied","Data":"ca5de28dbacf4de5d751779cbd52c8cd3b5de9d0166df83166efaaef5db49ff3"} Oct 30 04:10:03.728078 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.728052 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:10:03.796478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.796391 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\") pod \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " Oct 30 04:10:03.796478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.796438 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\") pod \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " Oct 30 04:10:03.796740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.796611 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\" (UniqueName: \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\") pod \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " Oct 30 04:10:03.796740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.796677 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\") pod \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\" (UID: \"85dff755-f0bf-49b5-b9bd-5c6fc649154f\") " Oct 30 04:10:03.796904 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.796879 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca" pod "85dff755-f0bf-49b5-b9bd-5c6fc649154f" (UID: "85dff755-f0bf-49b5-b9bd-5c6fc649154f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:03.797160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.797136 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85dff755-f0bf-49b5-b9bd-5c6fc649154f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:03.798986 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.798952 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls" pod "85dff755-f0bf-49b5-b9bd-5c6fc649154f" (UID: "85dff755-f0bf-49b5-b9bd-5c6fc649154f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:03.799090 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.799000 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2" pod "85dff755-f0bf-49b5-b9bd-5c6fc649154f" (UID: "85dff755-f0bf-49b5-b9bd-5c6fc649154f"). InnerVolumeSpecName "kube-api-access-nh5q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:03.799090 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.799080 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs" pod "85dff755-f0bf-49b5-b9bd-5c6fc649154f" (UID: "85dff755-f0bf-49b5-b9bd-5c6fc649154f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:03.898102 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.898068 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:03.898102 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.898097 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh5q2\" (UniqueName: \"kubernetes.io/projected/85dff755-f0bf-49b5-b9bd-5c6fc649154f-kube-api-access-nh5q2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:03.898102 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:03.898109 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85dff755-f0bf-49b5-b9bd-5c6fc649154f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:04.411424 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.411338 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" containerID="cri-o://9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b" gracePeriod=25 Oct 30 04:10:04.445140 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.445094 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" containerID="cri-o://fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8" gracePeriod=25 Oct 30 04:10:04.667987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.667908 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" Oct 30 04:10:04.667987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.667946 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2" event={"ID":"85dff755-f0bf-49b5-b9bd-5c6fc649154f","Type":"ContainerDied","Data":"0374d29b7fc3473254b203b5b062ba7383dc469a1d420d6795692ae048e5de6b"} Oct 30 04:10:04.668205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.668010 4560 scope.go:122] "RemoveContainer" containerID="ca5de28dbacf4de5d751779cbd52c8cd3b5de9d0166df83166efaaef5db49ff3" Oct 30 04:10:04.723513 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.723487 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:10:04.736314 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:04.736284 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-dd81fd-6cb74d7b75-gmtl2"] Oct 30 04:10:06.151400 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:06.151370 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" path="/var/lib/kubelet/pods/85dff755-f0bf-49b5-b9bd-5c6fc649154f/volumes" Oct 30 04:10:06.951493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:06.945132 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:06.951493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:06.946032 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:11.749353 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:11.749314 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:10:13.309013 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:13.308977 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 04:10:16.944995 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:16.944953 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:16.945412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:16.945033 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:17.132034 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.131991 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 04:10:17.132239 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.132200 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" podUID="68e2d3b3-174f-4524-a2f1-934813a0655a" containerName="maas-api" containerID="cri-o://1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af" gracePeriod=30 Oct 30 04:10:17.399673 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.399644 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 04:10:17.448760 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.448675 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\") pod \"68e2d3b3-174f-4524-a2f1-934813a0655a\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " Oct 30 04:10:17.448931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.448784 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\") pod \"68e2d3b3-174f-4524-a2f1-934813a0655a\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " Oct 30 04:10:17.448931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.448916 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\") pod \"68e2d3b3-174f-4524-a2f1-934813a0655a\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " Oct 30 04:10:17.449052 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.448960 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\" (UniqueName: \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\") pod \"68e2d3b3-174f-4524-a2f1-934813a0655a\" (UID: \"68e2d3b3-174f-4524-a2f1-934813a0655a\") " Oct 30 04:10:17.449478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.449310 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca" pod "68e2d3b3-174f-4524-a2f1-934813a0655a" (UID: "68e2d3b3-174f-4524-a2f1-934813a0655a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:17.451367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.451333 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls" pod "68e2d3b3-174f-4524-a2f1-934813a0655a" (UID: "68e2d3b3-174f-4524-a2f1-934813a0655a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:17.451657 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.451627 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq" pod "68e2d3b3-174f-4524-a2f1-934813a0655a" (UID: "68e2d3b3-174f-4524-a2f1-934813a0655a"). InnerVolumeSpecName "kube-api-access-xx4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:17.451808 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.451786 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs" pod "68e2d3b3-174f-4524-a2f1-934813a0655a" (UID: "68e2d3b3-174f-4524-a2f1-934813a0655a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:17.551091 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.551029 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2d3b3-174f-4524-a2f1-934813a0655a-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:17.551091 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.551072 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx4mq\" (UniqueName: \"kubernetes.io/projected/68e2d3b3-174f-4524-a2f1-934813a0655a-kube-api-access-xx4mq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:17.551091 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.551091 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:17.551665 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.551105 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e2d3b3-174f-4524-a2f1-934813a0655a-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:17.738693 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.738658 4560 generic.go:356] "Generic (PLEG): container finished" podID="68e2d3b3-174f-4524-a2f1-934813a0655a" containerID="1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af" exitCode=0 Oct 30 04:10:17.738874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.738711 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" Oct 30 04:10:17.738874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.738707 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" event={"ID":"68e2d3b3-174f-4524-a2f1-934813a0655a","Type":"ContainerDied","Data":"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af"} Oct 30 04:10:17.738874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.738834 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42" event={"ID":"68e2d3b3-174f-4524-a2f1-934813a0655a","Type":"ContainerDied","Data":"2407939a8e63cf87aab2223c8a1c63b8336b286d121f5873f08a07794a91f804"} Oct 30 04:10:17.738874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.738856 4560 scope.go:122] "RemoveContainer" containerID="1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af" Oct 30 04:10:17.748745 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.748718 4560 scope.go:122] "RemoveContainer" containerID="1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af" Oct 30 04:10:17.749032 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:17.749012 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af\": container with ID starting with 1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af not found: ID does not exist" containerID="1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af" Oct 30 04:10:17.749105 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.749041 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af"} err="failed to get container status \"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af\": rpc error: code = NotFound desc = could not find container \"1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af\": container with ID starting with 1ab604d61c2777ce81614d5c1775857d13ba1325b6aeeee422d2e74e61d3c9af not found: ID does not exist" Oct 30 04:10:17.790651 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.790620 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 04:10:17.805073 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:17.805034 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-398c25-7dc5dc4d5f-nbc42"] Oct 30 04:10:18.152477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:18.152397 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e2d3b3-174f-4524-a2f1-934813a0655a" path="/var/lib/kubelet/pods/68e2d3b3-174f-4524-a2f1-934813a0655a/volumes" Oct 30 04:10:18.310191 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:18.310138 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" containerID="cri-o://1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113" gracePeriod=25 Oct 30 04:10:24.609309 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.609275 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:24.703054 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.703027 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:10:24.727890 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.727864 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:10:24.775409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.775378 4560 generic.go:356] "Generic (PLEG): container finished" podID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerID="fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8" exitCode=0 Oct 30 04:10:24.775629 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.775426 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" Oct 30 04:10:24.775629 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.775459 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" event={"ID":"c6057a5a-55ad-4196-9b6d-7c881c04f59f","Type":"ContainerDied","Data":"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8"} Oct 30 04:10:24.775629 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.775552 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk" event={"ID":"c6057a5a-55ad-4196-9b6d-7c881c04f59f","Type":"ContainerDied","Data":"e63d78f0bb3fc3296202fe8f05610838d0da2e72dc26f0fcad1696299f54fcab"} Oct 30 04:10:24.775629 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.775585 4560 scope.go:122] "RemoveContainer" containerID="fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8" Oct 30 04:10:24.777084 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.777062 4560 generic.go:356] "Generic (PLEG): container finished" podID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerID="9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b" exitCode=0 Oct 30 04:10:24.777205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.777087 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" event={"ID":"7cd68ece-1b4d-4003-b82c-5064e7c3458c","Type":"ContainerDied","Data":"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b"} Oct 30 04:10:24.777205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.777118 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" Oct 30 04:10:24.777205 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.777134 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh" event={"ID":"7cd68ece-1b4d-4003-b82c-5064e7c3458c","Type":"ContainerDied","Data":"f18c674d90e18618179fa94afd6a32926be6642d82e858bf493bb110f18dbccb"} Oct 30 04:10:24.785787 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.785763 4560 scope.go:122] "RemoveContainer" containerID="fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8" Oct 30 04:10:24.786120 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:24.786097 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8\": container with ID starting with fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8 not found: ID does not exist" containerID="fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8" Oct 30 04:10:24.786243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.786136 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8"} err="failed to get container status \"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8\": rpc error: code = NotFound desc = could not find container \"fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8\": container with ID starting with fbcfb6b40121420ebe8533cac9b41801b596eb0fafbf9c3780e8cd55387536b8 not found: ID does not exist" Oct 30 04:10:24.786243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.786159 4560 scope.go:122] "RemoveContainer" containerID="9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b" Oct 30 04:10:24.796878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.796852 4560 scope.go:122] "RemoveContainer" containerID="9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b" Oct 30 04:10:24.797181 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:24.797157 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b\": container with ID starting with 9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b not found: ID does not exist" containerID="9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b" Oct 30 04:10:24.797266 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.797188 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b"} err="failed to get container status \"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b\": rpc error: code = NotFound desc = could not find container \"9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b\": container with ID starting with 9d9ff9f4529e1898fd80dab20bc79d71d7e89da6e7ed821fd0605f3ece982c5b not found: ID does not exist" Oct 30 04:10:24.827706 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.827670 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\" (UniqueName: \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\") pod \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " Oct 30 04:10:24.827871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.827712 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\" (UniqueName: \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\") pod \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " Oct 30 04:10:24.827871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.827790 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\") pod \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\" (UID: \"c6057a5a-55ad-4196-9b6d-7c881c04f59f\") " Oct 30 04:10:24.827871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.827825 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\") pod \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\" (UID: \"7cd68ece-1b4d-4003-b82c-5064e7c3458c\") " Oct 30 04:10:24.828218 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.828185 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume" pod "c6057a5a-55ad-4196-9b6d-7c881c04f59f" (UID: "c6057a5a-55ad-4196-9b6d-7c881c04f59f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:24.828313 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.828227 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume" pod "7cd68ece-1b4d-4003-b82c-5064e7c3458c" (UID: "7cd68ece-1b4d-4003-b82c-5064e7c3458c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:24.829940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.829916 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6" pod "7cd68ece-1b4d-4003-b82c-5064e7c3458c" (UID: "7cd68ece-1b4d-4003-b82c-5064e7c3458c"). InnerVolumeSpecName "kube-api-access-n8rz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:24.830058 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.829968 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r" pod "c6057a5a-55ad-4196-9b6d-7c881c04f59f" (UID: "c6057a5a-55ad-4196-9b6d-7c881c04f59f"). InnerVolumeSpecName "kube-api-access-frl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:24.929712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.929633 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frl2r\" (UniqueName: \"kubernetes.io/projected/c6057a5a-55ad-4196-9b6d-7c881c04f59f-kube-api-access-frl2r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:24.929712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.929663 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rz6\" (UniqueName: \"kubernetes.io/projected/7cd68ece-1b4d-4003-b82c-5064e7c3458c-kube-api-access-n8rz6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:24.929712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.929674 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6057a5a-55ad-4196-9b6d-7c881c04f59f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:24.929712 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:24.929683 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd68ece-1b4d-4003-b82c-5064e7c3458c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:25.125580 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:25.125545 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:10:25.128698 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:25.128673 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd81fd-7c65d94f47-b52gk"] Oct 30 04:10:25.162866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:25.162832 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:10:25.167569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:25.167533 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd81fd-84bc6865d8-4w7vh"] Oct 30 04:10:26.151538 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:26.151489 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" path="/var/lib/kubelet/pods/7cd68ece-1b4d-4003-b82c-5064e7c3458c/volumes" Oct 30 04:10:26.151920 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:26.151905 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" path="/var/lib/kubelet/pods/c6057a5a-55ad-4196-9b6d-7c881c04f59f/volumes" Oct 30 04:10:28.637801 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:28.637762 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 04:10:31.227139 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.227106 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 04:10:31.227751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.227303 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" podUID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" containerName="maas-api" containerID="cri-o://97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67" gracePeriod=30 Oct 30 04:10:31.498633 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.498606 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 04:10:31.599437 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.599401 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\") pod \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " Oct 30 04:10:31.599661 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.599584 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\") pod \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " Oct 30 04:10:31.599737 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.599665 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\") pod \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " Oct 30 04:10:31.599737 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.599720 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\" (UniqueName: \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\") pod \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\" (UID: \"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9\") " Oct 30 04:10:31.600061 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.600035 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca" pod "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" (UID: "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:31.601601 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.601575 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls" pod "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" (UID: "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:31.601830 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.601815 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs" pod "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" (UID: "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:31.601896 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.601820 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz" pod "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" (UID: "2c27b0e8-d70a-44ed-93db-0aed47f6c3a9"). InnerVolumeSpecName "kube-api-access-48kxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:31.701446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.701411 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:31.701446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.701441 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:31.701446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.701452 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48kxz\" (UniqueName: \"kubernetes.io/projected/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-kube-api-access-48kxz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:31.701713 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.701461 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:31.815547 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.815430 4560 generic.go:356] "Generic (PLEG): container finished" podID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" containerID="97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67" exitCode=0 Oct 30 04:10:31.815547 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.815483 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" Oct 30 04:10:31.815547 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.815512 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" event={"ID":"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9","Type":"ContainerDied","Data":"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67"} Oct 30 04:10:31.815830 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.815588 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8" event={"ID":"2c27b0e8-d70a-44ed-93db-0aed47f6c3a9","Type":"ContainerDied","Data":"44285fa04dcde0a893dba053b200e6f158d7383608a89446c257721bef8b8c35"} Oct 30 04:10:31.815830 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.815619 4560 scope.go:122] "RemoveContainer" containerID="97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67" Oct 30 04:10:31.825979 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.825952 4560 scope.go:122] "RemoveContainer" containerID="97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67" Oct 30 04:10:31.826228 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:31.826206 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67\": container with ID starting with 97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67 not found: ID does not exist" containerID="97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67" Oct 30 04:10:31.826278 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.826238 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67"} err="failed to get container status \"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67\": rpc error: code = NotFound desc = could not find container \"97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67\": container with ID starting with 97f0dff3844b60648c42c50bbf2802580c41262a9b48dac3f35ce1c2d5e21d67 not found: ID does not exist" Oct 30 04:10:31.875125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.875085 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 04:10:31.875342 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:31.875140 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-855e14-578498768-2zsv8"] Oct 30 04:10:32.152147 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:32.152071 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" path="/var/lib/kubelet/pods/2c27b0e8-d70a-44ed-93db-0aed47f6c3a9/volumes" Oct 30 04:10:33.639023 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:33.638981 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" containerID="cri-o://40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767" gracePeriod=25 Oct 30 04:10:34.608035 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:34.607994 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:37.058622 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:37.058579 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:37.290017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:37.289971 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 04:10:37.290250 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:37.290223 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" containerID="cri-o://0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5" gracePeriod=30 Oct 30 04:10:38.564184 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.564152 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 04:10:38.681697 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.681611 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\") pod \"02c721f6-07db-4192-9b56-cdb5b85cde3f\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " Oct 30 04:10:38.681853 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.681727 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\") pod \"02c721f6-07db-4192-9b56-cdb5b85cde3f\" (UID: \"02c721f6-07db-4192-9b56-cdb5b85cde3f\") " Oct 30 04:10:38.682145 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.682120 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume" pod "02c721f6-07db-4192-9b56-cdb5b85cde3f" (UID: "02c721f6-07db-4192-9b56-cdb5b85cde3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:38.683910 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.683881 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47" pod "02c721f6-07db-4192-9b56-cdb5b85cde3f" (UID: "02c721f6-07db-4192-9b56-cdb5b85cde3f"). InnerVolumeSpecName "kube-api-access-rhh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:38.783025 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.782991 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/02c721f6-07db-4192-9b56-cdb5b85cde3f-kube-api-access-rhh47\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:38.783025 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.783020 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c721f6-07db-4192-9b56-cdb5b85cde3f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:38.855202 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.855167 4560 generic.go:356] "Generic (PLEG): container finished" podID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerID="1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113" exitCode=0 Oct 30 04:10:38.855367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.855211 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" Oct 30 04:10:38.855367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.855227 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" event={"ID":"02c721f6-07db-4192-9b56-cdb5b85cde3f","Type":"ContainerDied","Data":"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113"} Oct 30 04:10:38.855367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.855283 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc" event={"ID":"02c721f6-07db-4192-9b56-cdb5b85cde3f","Type":"ContainerDied","Data":"415e1f8c213d406cd3023e3f4975f7e89909379500e6ef303e860f63a053b5ae"} Oct 30 04:10:38.855367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.855305 4560 scope.go:122] "RemoveContainer" containerID="1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113" Oct 30 04:10:38.865713 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.865499 4560 scope.go:122] "RemoveContainer" containerID="1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113" Oct 30 04:10:38.865892 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:38.865870 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113\": container with ID starting with 1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113 not found: ID does not exist" containerID="1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113" Oct 30 04:10:38.865962 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.865905 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113"} err="failed to get container status \"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113\": rpc error: code = NotFound desc = could not find container \"1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113\": container with ID starting with 1de60ef20281de80b7e7ed98dffcf5b4d2cd1ae652d198e80281a17101b50113 not found: ID does not exist" Oct 30 04:10:38.907153 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.907122 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 04:10:38.910374 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:38.910350 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-398c25-667cd69dc9-r8sxc"] Oct 30 04:10:40.151600 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:40.151563 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" path="/var/lib/kubelet/pods/02c721f6-07db-4192-9b56-cdb5b85cde3f/volumes" Oct 30 04:10:41.753258 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:41.753220 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:10:42.545598 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.545509 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 04:10:42.722121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722086 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722123 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722157 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722194 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722246 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722305 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722335 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722388 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722385 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722414 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\") pod \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\" (UID: \"7554b9fe-6f88-48a3-bc9d-4891493ad5a9\") " Oct 30 04:10:42.722773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722701 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:42.722883 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722795 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:42.723015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.722988 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:42.723094 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723018 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:42.723094 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723065 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:42.723214 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723163 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.723214 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723183 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.723214 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723197 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.723214 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723211 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.723442 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.723224 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.724804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.724770 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:10:42.724804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.724785 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:42.724804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.724776 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:42.725010 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.724799 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d" pod "7554b9fe-6f88-48a3-bc9d-4891493ad5a9" (UID: "7554b9fe-6f88-48a3-bc9d-4891493ad5a9"). InnerVolumeSpecName "kube-api-access-d6g2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:42.823999 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.823959 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.823999 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.823989 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.823999 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.824000 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6g2d\" (UniqueName: \"kubernetes.io/projected/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-kube-api-access-d6g2d\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.824444 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.824010 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7554b9fe-6f88-48a3-bc9d-4891493ad5a9-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:42.836377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.836343 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 04:10:42.844797 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.844764 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 04:10:42.879311 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.879279 4560 generic.go:356] "Generic (PLEG): container finished" podID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerID="0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5" exitCode=0 Oct 30 04:10:42.879497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.879323 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" Oct 30 04:10:42.879497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.879351 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" event={"ID":"7554b9fe-6f88-48a3-bc9d-4891493ad5a9","Type":"ContainerDied","Data":"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5"} Oct 30 04:10:42.879497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.879404 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv" event={"ID":"7554b9fe-6f88-48a3-bc9d-4891493ad5a9","Type":"ContainerDied","Data":"f45cf3e4a8b3f2c2b30292144fa89006b1f9cb996876a38d35804b61f4834f89"} Oct 30 04:10:42.879497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.879426 4560 scope.go:122] "RemoveContainer" containerID="0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5" Oct 30 04:10:42.889454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.889437 4560 scope.go:122] "RemoveContainer" containerID="0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5" Oct 30 04:10:42.889759 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:42.889739 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5\": container with ID starting with 0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5 not found: ID does not exist" containerID="0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5" Oct 30 04:10:42.889827 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.889771 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5"} err="failed to get container status \"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5\": rpc error: code = NotFound desc = could not find container \"0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5\": container with ID starting with 0f82f38fde7b2ead96aefdd05884f8a72dccb8f3d5a97be1fd0ece7a0233e3e5 not found: ID does not exist" Oct 30 04:10:42.955597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.955556 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 04:10:42.963042 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:42.963014 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pwxwv"] Oct 30 04:10:44.151718 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:44.151679 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" path="/var/lib/kubelet/pods/7554b9fe-6f88-48a3-bc9d-4891493ad5a9/volumes" Oct 30 04:10:47.058850 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:47.058813 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:53.896329 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.896257 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 04:10:53.924866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.924829 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\" (UniqueName: \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\") pod \"fceba5f9-ae00-4ea7-9fff-216955705e84\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " Oct 30 04:10:53.925038 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.924893 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\") pod \"fceba5f9-ae00-4ea7-9fff-216955705e84\" (UID: \"fceba5f9-ae00-4ea7-9fff-216955705e84\") " Oct 30 04:10:53.925221 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.925196 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume" pod "fceba5f9-ae00-4ea7-9fff-216955705e84" (UID: "fceba5f9-ae00-4ea7-9fff-216955705e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:53.926996 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.926969 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r" pod "fceba5f9-ae00-4ea7-9fff-216955705e84" (UID: "fceba5f9-ae00-4ea7-9fff-216955705e84"). InnerVolumeSpecName "kube-api-access-2rw9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:53.941800 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.941764 4560 generic.go:356] "Generic (PLEG): container finished" podID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerID="40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767" exitCode=0 Oct 30 04:10:53.941981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.941824 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" Oct 30 04:10:53.941981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.941856 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" event={"ID":"fceba5f9-ae00-4ea7-9fff-216955705e84","Type":"ContainerDied","Data":"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767"} Oct 30 04:10:53.941981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.941920 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6" event={"ID":"fceba5f9-ae00-4ea7-9fff-216955705e84","Type":"ContainerDied","Data":"fd71d460761732d6bcc7fdd422e1f95daf945f2038dc75361283b2d348a461b6"} Oct 30 04:10:53.941981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.941959 4560 scope.go:122] "RemoveContainer" containerID="40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767" Oct 30 04:10:53.953197 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.953173 4560 scope.go:122] "RemoveContainer" containerID="40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767" Oct 30 04:10:53.953538 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:53.953502 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767\": container with ID starting with 40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767 not found: ID does not exist" containerID="40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767" Oct 30 04:10:53.953646 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.953551 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767"} err="failed to get container status \"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767\": rpc error: code = NotFound desc = could not find container \"40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767\": container with ID starting with 40e740288b53ef75333d80ac3bae3698c640c0127f06f652ac8a92d04ef6f767 not found: ID does not exist" Oct 30 04:10:53.994194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:53.994159 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 04:10:54.000548 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:54.000496 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-855e14-666bcc4f5-r6nz6"] Oct 30 04:10:54.026116 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:54.026079 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rw9r\" (UniqueName: \"kubernetes.io/projected/fceba5f9-ae00-4ea7-9fff-216955705e84-kube-api-access-2rw9r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:54.026116 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:54.026112 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fceba5f9-ae00-4ea7-9fff-216955705e84-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:54.150673 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:54.150584 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" path="/var/lib/kubelet/pods/fceba5f9-ae00-4ea7-9fff-216955705e84/volumes" Oct 30 04:10:57.857404 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:57.857355 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" podUID="66840185-4102-4042-a065-d3a251a039d2" containerName="main" containerID="cri-o://29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13" gracePeriod=45 Oct 30 04:10:57.864317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:57.864273 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" podUID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" containerName="main" containerID="cri-o://a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca" gracePeriod=45 Oct 30 04:10:58.115195 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.115132 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 04:10:58.143925 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.143893 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 04:10:58.163955 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.163924 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.163980 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\" (UniqueName: \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164028 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164137 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164122 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164332 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164233 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.164413 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164393 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.164479 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164446 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164566 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164492 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.164759 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164568 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\") pod \"66840185-4102-4042-a065-d3a251a039d2\" (UID: \"66840185-4102-4042-a065-d3a251a039d2\") " Oct 30 04:10:58.164759 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164600 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.164759 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164672 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.164759 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164735 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.165004 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.164783 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.165084 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.165060 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.165751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.165314 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.165751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.165380 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.165751 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.165441 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.166401 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166078 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\") pod \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\" (UID: \"ba7a218a-b2e1-4687-8f6c-ae21f916004f\") " Oct 30 04:10:58.166903 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166881 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166906 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166918 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166929 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166940 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.166953 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.167328 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.167217 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "kube-api-access-9l5jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:58.167692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.167665 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.167692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.167685 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:58.168201 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.168180 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "kube-api-access-4bwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:58.168725 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.168707 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs" pod "ba7a218a-b2e1-4687-8f6c-ae21f916004f" (UID: "ba7a218a-b2e1-4687-8f6c-ae21f916004f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:58.169104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.169087 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm" pod "66840185-4102-4042-a065-d3a251a039d2" (UID: "66840185-4102-4042-a065-d3a251a039d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:58.267921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267873 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l5jv\" (UniqueName: \"kubernetes.io/projected/66840185-4102-4042-a065-d3a251a039d2-kube-api-access-9l5jv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.267921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267913 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840185-4102-4042-a065-d3a251a039d2-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.267921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267928 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840185-4102-4042-a065-d3a251a039d2-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.268178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267941 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/ba7a218a-b2e1-4687-8f6c-ae21f916004f-kube-api-access-4bwbp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.268178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267953 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7a218a-b2e1-4687-8f6c-ae21f916004f-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.268178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.267965 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7a218a-b2e1-4687-8f6c-ae21f916004f-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:10:58.972822 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.972785 4560 generic.go:356] "Generic (PLEG): container finished" podID="66840185-4102-4042-a065-d3a251a039d2" containerID="29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13" exitCode=0 Oct 30 04:10:58.973305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.972856 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" Oct 30 04:10:58.973305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.972872 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" event={"ID":"66840185-4102-4042-a065-d3a251a039d2","Type":"ContainerDied","Data":"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13"} Oct 30 04:10:58.973305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.972930 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj" event={"ID":"66840185-4102-4042-a065-d3a251a039d2","Type":"ContainerDied","Data":"81b2450f9226f1c16f3cf4f84f1f04945c39397b20fa8e2d7ce7b484029ef1f7"} Oct 30 04:10:58.973305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.972954 4560 scope.go:122] "RemoveContainer" containerID="29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13" Oct 30 04:10:58.974357 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.974332 4560 generic.go:356] "Generic (PLEG): container finished" podID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" containerID="a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca" exitCode=0 Oct 30 04:10:58.974446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.974361 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" event={"ID":"ba7a218a-b2e1-4687-8f6c-ae21f916004f","Type":"ContainerDied","Data":"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca"} Oct 30 04:10:58.974446 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.974412 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" event={"ID":"ba7a218a-b2e1-4687-8f6c-ae21f916004f","Type":"ContainerDied","Data":"8b4fa1c9978db6d303d37379f36dde3b8a03d6ebdea576afb97dab6bc4c71b88"} Oct 30 04:10:58.974611 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.974438 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd" Oct 30 04:10:58.982161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.982140 4560 scope.go:122] "RemoveContainer" containerID="29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13" Oct 30 04:10:58.982499 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:58.982469 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13\": container with ID starting with 29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13 not found: ID does not exist" containerID="29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13" Oct 30 04:10:58.982612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.982525 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13"} err="failed to get container status \"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13\": rpc error: code = NotFound desc = could not find container \"29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13\": container with ID starting with 29adaf9e5705c4ee696a873fcc5c30668b4a3dfd31980f0ece120201d6addb13 not found: ID does not exist" Oct 30 04:10:58.982612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.982553 4560 scope.go:122] "RemoveContainer" containerID="a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca" Oct 30 04:10:58.991763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.991744 4560 scope.go:122] "RemoveContainer" containerID="a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca" Oct 30 04:10:58.992035 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:10:58.992014 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca\": container with ID starting with a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca not found: ID does not exist" containerID="a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca" Oct 30 04:10:58.992102 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:58.992043 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca"} err="failed to get container status \"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca\": rpc error: code = NotFound desc = could not find container \"a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca\": container with ID starting with a5b2af8d8e796c70ac75f7ccb00c04fa7e59d02e2ffbdd4f5119fd424f9d76ca not found: ID does not exist" Oct 30 04:10:59.024305 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:59.024270 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 04:10:59.037232 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:59.037192 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/simulator-w2-2d5b09-kserve-7689b5f567-krmcd"] Oct 30 04:10:59.061966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:59.061932 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 04:10:59.066569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:10:59.066542 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/premium-w2-2d5b09-kserve-5b8ff957fd-hgbhj"] Oct 30 04:11:00.151291 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:00.151255 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66840185-4102-4042-a065-d3a251a039d2" path="/var/lib/kubelet/pods/66840185-4102-4042-a065-d3a251a039d2/volumes" Oct 30 04:11:00.151747 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:00.151732 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" path="/var/lib/kubelet/pods/ba7a218a-b2e1-4687-8f6c-ae21f916004f/volumes" Oct 30 04:11:07.059550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:07.059450 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:11:07.290567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:07.290509 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:11:07.541239 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:07.541206 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:11:09.460561 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:11:09.460502 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:11:11.749310 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:11:11.749274 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:11:41.749422 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:11:41.749381 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:12:11.749653 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:12:11.749618 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:12:41.749675 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:12:41.749587 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:13:11.748779 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:13:11.748739 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:13:41.748216 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:13:41.748169 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:14:03.606671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:03.606626 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:14:03.607329 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:03.607292 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:14:03.771731 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:03.771687 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:14:03.772053 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:03.772018 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:14:11.749616 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:14:11.749573 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:14:41.749385 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:14:41.749324 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:14:48.783681 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:48.783615 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:14:48.784488 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:48.784452 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:14:49.100597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:49.099638 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:14:49.100597 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:49.099746 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:14:53.737894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.737851 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk"] Oct 30 04:14:53.738330 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738316 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" containerName="maas-api" Oct 30 04:14:53.738377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738332 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" containerName="maas-api" Oct 30 04:14:53.738377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738351 4560 cpu_manager.go:414] "containerMap: removing container" podUID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" containerName="maas-api" Oct 30 04:14:53.738377 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738357 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" containerName="maas-api" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738379 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738387 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738396 4560 cpu_manager.go:414] "containerMap: removing container" podUID="66840185-4102-4042-a065-d3a251a039d2" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738401 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66840185-4102-4042-a065-d3a251a039d2" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738409 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738417 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738426 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738431 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738439 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738443 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738449 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738454 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" containerName="main" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738461 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" containerName="maas-api" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738465 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" containerName="maas-api" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738472 4560 cpu_manager.go:414] "containerMap: removing container" podUID="68e2d3b3-174f-4524-a2f1-934813a0655a" containerName="maas-api" Oct 30 04:14:53.738477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738476 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e2d3b3-174f-4524-a2f1-934813a0655a" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738487 4560 cpu_manager.go:414] "containerMap: removing container" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738494 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738506 4560 cpu_manager.go:414] "containerMap: removing container" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738511 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738530 4560 cpu_manager.go:414] "containerMap: removing container" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738535 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738542 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738547 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738552 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7cc218d5-1147-4e69-a7a1-618b7b33e489" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738556 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc218d5-1147-4e69-a7a1-618b7b33e489" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738572 4560 cpu_manager.go:414] "containerMap: removing container" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738579 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738588 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738592 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738598 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738603 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738618 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738622 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738631 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c937282b-846b-455f-8be1-f63d060653ac" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738636 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c937282b-846b-455f-8be1-f63d060653ac" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738701 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="61014647-b515-45f6-886c-87b102274c29" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738711 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c721f6-07db-4192-9b56-cdb5b85cde3f" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738717 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6ccbee-83a6-4bad-a865-21018fa6e9fa" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738722 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceba5f9-ae00-4ea7-9fff-216955705e84" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738734 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc218d5-1147-4e69-a7a1-618b7b33e489" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738743 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7554b9fe-6f88-48a3-bc9d-4891493ad5a9" containerName="istio-proxy" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738749 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee2fe47-8e05-4edc-b0c2-b15bdac78880" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738754 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d833a78d-8e47-45ea-9e16-0b503352a97c" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738761 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c27b0e8-d70a-44ed-93db-0aed47f6c3a9" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738766 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c937282b-846b-455f-8be1-f63d060653ac" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738771 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cff3e2-50cd-4110-bf7a-2bebeb18f822" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738777 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dff755-f0bf-49b5-b9bd-5c6fc649154f" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738783 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6057a5a-55ad-4196-9b6d-7c881c04f59f" containerName="payload-pre-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738789 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="66840185-4102-4042-a065-d3a251a039d2" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738794 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9d1e24-9e81-47d0-81fc-85f9662c41bc" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738799 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd68ece-1b4d-4003-b82c-5064e7c3458c" containerName="payload-processing" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738806 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7a218a-b2e1-4687-8f6c-ae21f916004f" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738815 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e556d36d-074d-4f34-b96d-5c6b9b35dd77" containerName="main" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738821 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f8c88a-6b87-4e53-8b33-ed348f369c7b" containerName="maas-api" Oct 30 04:14:53.739056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.738827 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e2d3b3-174f-4524-a2f1-934813a0655a" containerName="maas-api" Oct 30 04:14:53.741981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.741966 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.760124 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.760097 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk"] Oct 30 04:14:53.770554 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770512 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:14:53.770554 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770532 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:14:53.770754 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770562 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:14:53.770754 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770625 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:14:53.770832 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770811 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.770879 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770866 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.770981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.770963 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8lw\" (UniqueName: \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.871895 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.871857 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.871895 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.871897 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8lw\" (UniqueName: \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.872179 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.871980 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.872605 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.872578 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.874292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.874270 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:53.891826 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:53.891793 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8lw\" (UniqueName: \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\") pod \"maas-api-revoke-sub-73626ff4-7hskk\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:54.054498 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:54.054410 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:54.352638 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:54.352605 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk"] Oct 30 04:14:54.354110 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:14:54.354077 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3947a951_3775_4de9_bb75_430002108124.slice/crio-e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d WatchSource:0}: Error finding container e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d: Status 404 returned error can't find the container with id e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d Oct 30 04:14:55.254757 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:55.254718 4560 generic.go:356] "Generic (PLEG): container finished" podID="3947a951-3775-4de9-bb75-430002108124" containerID="89ae7d06765cc8403bf8ea4c8404bd8549c04ad3cc2328ce47500187dd186973" exitCode=0 Oct 30 04:14:55.255155 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:55.254809 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" event={"ID":"3947a951-3775-4de9-bb75-430002108124","Type":"ContainerDied","Data":"89ae7d06765cc8403bf8ea4c8404bd8549c04ad3cc2328ce47500187dd186973"} Oct 30 04:14:55.255155 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:55.254893 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" event={"ID":"3947a951-3775-4de9-bb75-430002108124","Type":"ContainerStarted","Data":"e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d"} Oct 30 04:14:56.393867 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.393838 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:56.496444 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.496413 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\") pod \"3947a951-3775-4de9-bb75-430002108124\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " Oct 30 04:14:56.496682 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.496486 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\" (UniqueName: \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\") pod \"3947a951-3775-4de9-bb75-430002108124\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " Oct 30 04:14:56.496682 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.496583 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\") pod \"3947a951-3775-4de9-bb75-430002108124\" (UID: \"3947a951-3775-4de9-bb75-430002108124\") " Oct 30 04:14:56.496835 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.496809 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca" pod "3947a951-3775-4de9-bb75-430002108124" (UID: "3947a951-3775-4de9-bb75-430002108124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:14:56.497016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.496994 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3947a951-3775-4de9-bb75-430002108124-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:14:56.498656 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.498627 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca" pod "3947a951-3775-4de9-bb75-430002108124" (UID: "3947a951-3775-4de9-bb75-430002108124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:14:56.498656 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.498647 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw" pod "3947a951-3775-4de9-bb75-430002108124" (UID: "3947a951-3775-4de9-bb75-430002108124"). InnerVolumeSpecName "kube-api-access-ml8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:14:56.597918 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.597825 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8lw\" (UniqueName: \"kubernetes.io/projected/3947a951-3775-4de9-bb75-430002108124-kube-api-access-ml8lw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:14:56.597918 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:56.597859 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3947a951-3775-4de9-bb75-430002108124-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:14:57.266739 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:57.266706 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" Oct 30 04:14:57.266921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:57.266705 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk" event={"ID":"3947a951-3775-4de9-bb75-430002108124","Type":"ContainerDied","Data":"e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d"} Oct 30 04:14:57.266921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:14:57.266837 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75b0efe81826849d4c24214df3daa1bdac12edcb94897db6c95f59e9f76925d" Oct 30 04:15:00.140281 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.140248 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:00.140739 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.140725 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3947a951-3775-4de9-bb75-430002108124" containerName="revoke-keys" Oct 30 04:15:00.140804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.140741 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3947a951-3775-4de9-bb75-430002108124" containerName="revoke-keys" Oct 30 04:15:00.140877 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.140862 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3947a951-3775-4de9-bb75-430002108124" containerName="revoke-keys" Oct 30 04:15:00.144224 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.144206 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.174149 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.174117 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:15:00.174421 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.174117 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:15:00.174421 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.174232 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:15:00.176181 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.176152 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:00.235531 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.235482 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srn8j\" (UniqueName: \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.235697 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.235603 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.336743 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.336699 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn8j\" (UniqueName: \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.336924 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.336756 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.337466 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.337443 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.360256 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.360224 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn8j\" (UniqueName: \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\") pod \"maas-api-key-cleanup-29888895-bmg26\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.462200 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.462168 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:00.764043 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:00.764007 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:00.765672 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:15:00.765641 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f0f50c_259c_44a6_8db0_9fcc5341bc58.slice/crio-bb3404c8f4f4bd70819e8b7ba4dc4e6a1f7e3c0eebcd27a488ce450a30dd0307 WatchSource:0}: Error finding container bb3404c8f4f4bd70819e8b7ba4dc4e6a1f7e3c0eebcd27a488ce450a30dd0307: Status 404 returned error can't find the container with id bb3404c8f4f4bd70819e8b7ba4dc4e6a1f7e3c0eebcd27a488ce450a30dd0307 Oct 30 04:15:01.288162 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:01.288126 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerStarted","Data":"913ef9c724ea61cbb33649c4c757939c7c1d10af326f96a429bd8fa2d0ec57c2"} Oct 30 04:15:01.288672 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:01.288186 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerStarted","Data":"bb3404c8f4f4bd70819e8b7ba4dc4e6a1f7e3c0eebcd27a488ce450a30dd0307"} Oct 30 04:15:01.327940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:01.327823 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" podStartSLOduration=1.327808675 podStartE2EDuration="1.327808675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:01.326224119 +0000 UTC m=+4329.311344008" watchObservedRunningTime="2026-10-30 04:15:01.327808675 +0000 UTC m=+4329.312928562" Oct 30 04:15:02.294416 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:02.294373 4560 generic.go:356] "Generic (PLEG): container finished" podID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerID="913ef9c724ea61cbb33649c4c757939c7c1d10af326f96a429bd8fa2d0ec57c2" exitCode=60 Oct 30 04:15:02.294878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:02.294461 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerDied","Data":"913ef9c724ea61cbb33649c4c757939c7c1d10af326f96a429bd8fa2d0ec57c2"} Oct 30 04:15:02.294878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:02.294707 4560 scope.go:122] "RemoveContainer" containerID="913ef9c724ea61cbb33649c4c757939c7c1d10af326f96a429bd8fa2d0ec57c2" Oct 30 04:15:03.301364 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:03.301328 4560 generic.go:356] "Generic (PLEG): container finished" podID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerID="4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff" exitCode=60 Oct 30 04:15:03.301794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:03.301415 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerDied","Data":"4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff"} Oct 30 04:15:03.301794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:03.301469 4560 scope.go:122] "RemoveContainer" containerID="913ef9c724ea61cbb33649c4c757939c7c1d10af326f96a429bd8fa2d0ec57c2" Oct 30 04:15:03.301794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:03.301779 4560 scope.go:122] "RemoveContainer" containerID="4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff" Oct 30 04:15:03.302068 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:15:03.302040 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-bmg26_odh-ai-gateway-infra(d4f0f50c-259c-44a6-8db0-9fcc5341bc58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" Oct 30 04:15:04.308805 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:04.308778 4560 scope.go:122] "RemoveContainer" containerID="4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff" Oct 30 04:15:04.309169 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:15:04.309008 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-bmg26_odh-ai-gateway-infra(d4f0f50c-259c-44a6-8db0-9fcc5341bc58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" Oct 30 04:15:11.817961 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:15:11.817912 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:15:16.144919 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:16.144892 4560 scope.go:122] "RemoveContainer" containerID="4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff" Oct 30 04:15:17.375125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:17.375050 4560 generic.go:356] "Generic (PLEG): container finished" podID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerID="f31b6ecc6f4eae9e0a97ac8a4fbae306b977de8a0f161767a09e6466489af70b" exitCode=60 Oct 30 04:15:17.376937 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:17.375125 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerDied","Data":"f31b6ecc6f4eae9e0a97ac8a4fbae306b977de8a0f161767a09e6466489af70b"} Oct 30 04:15:17.376937 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:17.375332 4560 scope.go:122] "RemoveContainer" containerID="4c9f7f558fd9f00e4cdd4b308d6950af19b8dae6d2314712e841ea2490c0c5ff" Oct 30 04:15:17.376937 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:17.375934 4560 scope.go:122] "RemoveContainer" containerID="f31b6ecc6f4eae9e0a97ac8a4fbae306b977de8a0f161767a09e6466489af70b" Oct 30 04:15:17.376937 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:15:17.376165 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-bmg26_odh-ai-gateway-infra(d4f0f50c-259c-44a6-8db0-9fcc5341bc58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" Oct 30 04:15:18.427128 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.427091 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:18.630039 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.629991 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:15:18.630397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.630351 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:15:18.755623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.755590 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:18.782402 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.772661 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:15:18.782402 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.772740 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:15:18.847834 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.847793 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\") pod \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " Oct 30 04:15:18.848059 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.847885 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\" (UniqueName: \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\") pod \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\" (UID: \"d4f0f50c-259c-44a6-8db0-9fcc5341bc58\") " Oct 30 04:15:18.848307 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.848278 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca" pod "d4f0f50c-259c-44a6-8db0-9fcc5341bc58" (UID: "d4f0f50c-259c-44a6-8db0-9fcc5341bc58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:15:18.921325 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.921220 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j" pod "d4f0f50c-259c-44a6-8db0-9fcc5341bc58" (UID: "d4f0f50c-259c-44a6-8db0-9fcc5341bc58"). InnerVolumeSpecName "kube-api-access-srn8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:15:18.953292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.953211 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:15:18.953292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:18.953254 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srn8j\" (UniqueName: \"kubernetes.io/projected/d4f0f50c-259c-44a6-8db0-9fcc5341bc58-kube-api-access-srn8j\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:15:19.098843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.098734 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:15:19.099177 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.099141 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:15:19.410987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.410888 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" event={"ID":"d4f0f50c-259c-44a6-8db0-9fcc5341bc58","Type":"ContainerDied","Data":"bb3404c8f4f4bd70819e8b7ba4dc4e6a1f7e3c0eebcd27a488ce450a30dd0307"} Oct 30 04:15:19.410987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.410957 4560 scope.go:122] "RemoveContainer" containerID="f31b6ecc6f4eae9e0a97ac8a4fbae306b977de8a0f161767a09e6466489af70b" Oct 30 04:15:19.410987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.410909 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26" Oct 30 04:15:19.465527 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.465491 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:19.469125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:19.469099 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-bmg26"] Oct 30 04:15:20.151298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:20.151264 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" path="/var/lib/kubelet/pods/d4f0f50c-259c-44a6-8db0-9fcc5341bc58/volumes" Oct 30 04:15:41.750631 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:15:41.750576 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:15:42.336119 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336081 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:15:42.336745 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336720 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336747 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336791 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336811 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336801 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336924 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336914 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336933 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.336966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.336945 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:15:42.340596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.340572 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.370392 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.370358 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26a1d1\"/\"kube-root-ca.crt\"" Oct 30 04:15:42.370612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.370467 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26a1d1\"/\"openshift-service-ca.crt\"" Oct 30 04:15:42.370612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.370496 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26a1d1\"/\"simulator-single-main-26a1d1-kserve-self-signed-certs\"" Oct 30 04:15:42.370612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.370585 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26a1d1\"/\"default-dockercfg-p24qp\"" Oct 30 04:15:42.372846 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.372817 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:15:42.495133 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495085 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.495337 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495141 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkntt\" (UniqueName: \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.495337 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495255 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.495472 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495348 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.495472 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495372 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.495472 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.495407 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.596966 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkntt\" (UniqueName: \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597057 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597101 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597125 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597177 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597210 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597700 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597670 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597700 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597685 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.597903 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.597820 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.599484 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.599463 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.599974 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.599952 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.616443 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.616418 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkntt\" (UniqueName: \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\") pod \"simulator-single-main-26a1d1-kserve-86b978487-7rzpp\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.654382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.654347 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:42.954497 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:42.954460 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:15:42.956274 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:15:42.956247 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c9bca68_4a28_4c15_8793_9b8ecc27626b.slice/crio-1b841601aa0fb295f595f1bb163ff1d364969b8e2627489c069f45bdd7428962 WatchSource:0}: Error finding container 1b841601aa0fb295f595f1bb163ff1d364969b8e2627489c069f45bdd7428962: Status 404 returned error can't find the container with id 1b841601aa0fb295f595f1bb163ff1d364969b8e2627489c069f45bdd7428962 Oct 30 04:15:43.527322 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:43.527283 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" event={"ID":"9c9bca68-4a28-4c15-8793-9b8ecc27626b","Type":"ContainerStarted","Data":"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f"} Oct 30 04:15:43.527322 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:43.527327 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" event={"ID":"9c9bca68-4a28-4c15-8793-9b8ecc27626b","Type":"ContainerStarted","Data":"1b841601aa0fb295f595f1bb163ff1d364969b8e2627489c069f45bdd7428962"} Oct 30 04:15:43.569509 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:43.569453 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" podStartSLOduration=1.56943855 podStartE2EDuration="1.56943855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:43.567317547 +0000 UTC m=+4371.552437436" watchObservedRunningTime="2026-10-30 04:15:43.56943855 +0000 UTC m=+4371.554558438" Oct 30 04:15:43.654440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:43.654405 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:52.654570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:52.654512 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:52.667265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:52.667241 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:15:53.588748 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:15:53.588712 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:16:11.750080 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:16:11.750039 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:16:28.025640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:28.025601 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:16:41.749059 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:16:41.749020 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:16:43.045693 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.045640 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" podUID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" containerName="main" containerID="cri-o://dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f" gracePeriod=45 Oct 30 04:16:43.301570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.301495 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:16:43.416409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416369 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\" (UniqueName: \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416449 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416553 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416583 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416640 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416611 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416691 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\") pod \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\" (UID: \"9c9bca68-4a28-4c15-8793-9b8ecc27626b\") " Oct 30 04:16:43.416878 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416719 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:43.417000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416930 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:43.417000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.416994 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.417193 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.417164 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:43.419009 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.418984 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:43.419173 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.419150 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "kube-api-access-rkntt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:43.419280 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.419242 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs" pod "9c9bca68-4a28-4c15-8793-9b8ecc27626b" (UID: "9c9bca68-4a28-4c15-8793-9b8ecc27626b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:43.517738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.517699 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.517738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.517731 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.517738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.517741 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.517738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.517749 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9bca68-4a28-4c15-8793-9b8ecc27626b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.518020 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.517760 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkntt\" (UniqueName: \"kubernetes.io/projected/9c9bca68-4a28-4c15-8793-9b8ecc27626b-kube-api-access-rkntt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.816675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.816637 4560 generic.go:356] "Generic (PLEG): container finished" podID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" containerID="dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f" exitCode=0 Oct 30 04:16:43.816866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.816685 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" Oct 30 04:16:43.816866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.816729 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" event={"ID":"9c9bca68-4a28-4c15-8793-9b8ecc27626b","Type":"ContainerDied","Data":"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f"} Oct 30 04:16:43.816866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.816786 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp" event={"ID":"9c9bca68-4a28-4c15-8793-9b8ecc27626b","Type":"ContainerDied","Data":"1b841601aa0fb295f595f1bb163ff1d364969b8e2627489c069f45bdd7428962"} Oct 30 04:16:43.816866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.816809 4560 scope.go:122] "RemoveContainer" containerID="dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f" Oct 30 04:16:43.827397 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.827373 4560 scope.go:122] "RemoveContainer" containerID="dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f" Oct 30 04:16:43.827733 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:16:43.827711 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f\": container with ID starting with dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f not found: ID does not exist" containerID="dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f" Oct 30 04:16:43.827822 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.827745 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f"} err="failed to get container status \"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f\": rpc error: code = NotFound desc = could not find container \"dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f\": container with ID starting with dceeb006fc758d9e57f3addd1c13d7f7625ec97eb7ffd7732fb404672285596f not found: ID does not exist" Oct 30 04:16:43.869851 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.869812 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:16:43.882505 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:43.882464 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-26a1d1/simulator-single-main-26a1d1-kserve-86b978487-7rzpp"] Oct 30 04:16:44.151701 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:16:44.151609 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" path="/var/lib/kubelet/pods/9c9bca68-4a28-4c15-8793-9b8ecc27626b/volumes" Oct 30 04:17:03.599541 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:03.596627 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:17:03.599541 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:03.596720 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:17:03.775679 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:03.775625 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88m6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:17:03.775983 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:03.775950 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88m6n" podUID="0806baf1-a58a-48df-8c59-aabafc920d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:17:04.098542 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:04.097695 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lkn75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:17:04.098542 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:17:04.097810 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lkn75" podUID="dcf485c0-995a-4d10-ac7f-dd565faf112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:17:11.749824 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:17:11.749782 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:17:41.748760 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:17:41.748724 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:18:11.749288 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:18:11.749245 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:18:41.748431 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:18:41.748340 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:19:11.749751 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:19:11.749708 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:19:41.749040 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:19:41.749003 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:19:57.030405 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:19:57.030364 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk"] Oct 30 04:19:57.042999 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:19:57.042964 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73626ff4-7hskk"] Oct 30 04:19:58.154418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:19:58.154379 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3947a951-3775-4de9-bb75-430002108124" path="/var/lib/kubelet/pods/3947a951-3775-4de9-bb75-430002108124/volumes" Oct 30 04:20:11.749659 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:20:11.749579 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:20:41.749229 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:20:41.749191 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:21:11.750175 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:21:11.750131 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:21:41.809596 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:21:41.802718 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:21:53.944946 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:21:53.944914 4560 scope.go:122] "RemoveContainer" containerID="89ae7d06765cc8403bf8ea4c8404bd8549c04ad3cc2328ce47500187dd186973" Oct 30 04:22:11.748337 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:22:11.748299 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:22:41.748807 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:22:41.748766 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:23:11.754172 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:23:11.754089 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:23:41.749393 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:23:41.749350 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:24:11.748983 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:24:11.748942 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:24:41.749413 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:24:41.749321 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:25:11.749414 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:25:11.749371 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:25:41.749173 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:25:41.749129 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:26:11.748925 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:26:11.748843 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:26:41.750402 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:26:41.750357 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:27:11.748932 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:27:11.748893 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:27:41.749080 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:27:41.749001 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:28:11.820544 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:28:11.818712 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:28:26.641900 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.641860 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:26.642562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.642266 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" containerName="main" Oct 30 04:28:26.642562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.642279 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" containerName="main" Oct 30 04:28:26.642562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.642306 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:28:26.642562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.642312 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f0f50c-259c-44a6-8db0-9fcc5341bc58" containerName="cleanup" Oct 30 04:28:26.642562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.642388 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9bca68-4a28-4c15-8793-9b8ecc27626b" containerName="main" Oct 30 04:28:26.645582 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.645555 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.677297 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.677263 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:28:26.677493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.677319 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 04:28:26.677493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.677352 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:28:26.677644 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.677582 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 04:28:26.678234 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.678198 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8gxpp\"" Oct 30 04:28:26.692499 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.692472 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:26.758363 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.758323 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2fk\" (UniqueName: \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.758562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.758374 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.758562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.758482 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.758562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.758541 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.859257 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.859224 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.859480 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.859289 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.859480 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.859325 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.859480 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.859385 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2fk\" (UniqueName: \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.859989 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.859966 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.861853 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.861824 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.861985 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.861864 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.879894 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.879863 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2fk\" (UniqueName: \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\") pod \"maas-api-65b844bfb4-tjs2q\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:26.958930 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:26.958899 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:27.258312 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:27.258278 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:27.260234 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:28:27.260204 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f7af2a_22ab_4561_8cc4_88c73f15a5bd.slice/crio-4ee348331533ba948ee1a59f949a5442f0fd60281efdec3024e6e163bba8f587 WatchSource:0}: Error finding container 4ee348331533ba948ee1a59f949a5442f0fd60281efdec3024e6e163bba8f587: Status 404 returned error can't find the container with id 4ee348331533ba948ee1a59f949a5442f0fd60281efdec3024e6e163bba8f587 Oct 30 04:28:27.261589 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:27.261571 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:28:27.589912 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:27.589831 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" event={"ID":"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd","Type":"ContainerStarted","Data":"4ee348331533ba948ee1a59f949a5442f0fd60281efdec3024e6e163bba8f587"} Oct 30 04:28:28.595940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:28.595897 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" event={"ID":"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd","Type":"ContainerStarted","Data":"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5"} Oct 30 04:28:28.596449 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:28.595994 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:28.637473 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:28.637414 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" podStartSLOduration=2.165378468 podStartE2EDuration="2.637401518s" totalImagesPullingTime="472.02305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:28.635729702 +0000 UTC m=+5136.620849589" watchObservedRunningTime="2026-10-30 04:28:28.637401518 +0000 UTC m=+5136.622521419" Oct 30 04:28:34.606931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:34.606897 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:41.750769 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:28:41.750720 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:28:44.641069 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.641032 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:44.641664 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.641296 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" podUID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" containerName="maas-api" containerID="cri-o://e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5" gracePeriod=30 Oct 30 04:28:44.899421 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.899358 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:44.984861 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.984831 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\" (UniqueName: \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\") pod \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " Oct 30 04:28:44.985055 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.984877 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\") pod \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " Oct 30 04:28:44.985055 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.984969 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\") pod \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " Oct 30 04:28:44.985055 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.985031 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\") pod \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\" (UID: \"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd\") " Oct 30 04:28:44.985379 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.985354 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca" pod "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" (UID: "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:44.986955 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.986930 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk" pod "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" (UID: "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd"). InnerVolumeSpecName "kube-api-access-bx2fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:44.987055 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.986977 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs" pod "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" (UID: "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:44.987105 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:44.987068 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls" pod "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" (UID: "b9f7af2a-22ab-4561-8cc4-88c73f15a5bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:45.085913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.085863 4560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:45.085913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.085909 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:45.085913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.085918 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx2fk\" (UniqueName: \"kubernetes.io/projected/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-kube-api-access-bx2fk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:45.085913 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.085927 4560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:45.235354 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.235322 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg"] Oct 30 04:28:45.235773 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.235758 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" containerName="maas-api" Oct 30 04:28:45.235823 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.235775 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" containerName="maas-api" Oct 30 04:28:45.235863 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.235853 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" containerName="maas-api" Oct 30 04:28:45.239091 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.239075 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.257445 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.257413 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:28:45.257587 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.257508 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:28:45.259468 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.259427 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg"] Oct 30 04:28:45.389062 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.389018 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklrn\" (UniqueName: \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.389287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.389091 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.389287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.389182 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.489962 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.489851 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklrn\" (UniqueName: \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.489962 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.489946 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.490175 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.489970 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.490615 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.490590 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.492352 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.492327 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.509927 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.509906 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklrn\" (UniqueName: \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\") pod \"maas-api-revoke-sub-559dd0f5-4gvvg\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.552002 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.551969 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:45.735415 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.735382 4560 generic.go:356] "Generic (PLEG): container finished" podID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" containerID="e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5" exitCode=0 Oct 30 04:28:45.735886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.735428 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" event={"ID":"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd","Type":"ContainerDied","Data":"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5"} Oct 30 04:28:45.735886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.735433 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" Oct 30 04:28:45.735886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.735479 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q" event={"ID":"b9f7af2a-22ab-4561-8cc4-88c73f15a5bd","Type":"ContainerDied","Data":"4ee348331533ba948ee1a59f949a5442f0fd60281efdec3024e6e163bba8f587"} Oct 30 04:28:45.735886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.735501 4560 scope.go:122] "RemoveContainer" containerID="e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5" Oct 30 04:28:45.744462 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.744445 4560 scope.go:122] "RemoveContainer" containerID="e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5" Oct 30 04:28:45.744789 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:28:45.744770 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5\": container with ID starting with e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5 not found: ID does not exist" containerID="e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5" Oct 30 04:28:45.744867 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.744798 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5"} err="failed to get container status \"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5\": rpc error: code = NotFound desc = could not find container \"e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5\": container with ID starting with e3636f3fc652a0e99d6b5062b8e0987ef95c5da163bcf2bc495e1a06049d93f5 not found: ID does not exist" Oct 30 04:28:45.787572 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.787543 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:45.800512 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.800482 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-tjs2q"] Oct 30 04:28:45.850947 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:45.850914 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg"] Oct 30 04:28:45.852795 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:28:45.852672 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6118fafe_5cb0_41c5_a89a_38d664068d7d.slice/crio-546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339 WatchSource:0}: Error finding container 546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339: Status 404 returned error can't find the container with id 546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339 Oct 30 04:28:46.151647 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:46.151613 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f7af2a-22ab-4561-8cc4-88c73f15a5bd" path="/var/lib/kubelet/pods/b9f7af2a-22ab-4561-8cc4-88c73f15a5bd/volumes" Oct 30 04:28:46.742141 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:46.742076 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" event={"ID":"6118fafe-5cb0-41c5-a89a-38d664068d7d","Type":"ContainerStarted","Data":"40c09a27e2e396931ae61135b1080d24377a4b23f6174a15a584a20e93ecae9e"} Oct 30 04:28:46.742645 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:46.742154 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" event={"ID":"6118fafe-5cb0-41c5-a89a-38d664068d7d","Type":"ContainerStarted","Data":"546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339"} Oct 30 04:28:46.784114 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:46.784067 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" podStartSLOduration=1.7840530380000001 podStartE2EDuration="1.784053038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:46.782919619 +0000 UTC m=+5154.768039509" watchObservedRunningTime="2026-10-30 04:28:46.784053038 +0000 UTC m=+5154.769172950" Oct 30 04:28:47.748617 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:47.748578 4560 generic.go:356] "Generic (PLEG): container finished" podID="6118fafe-5cb0-41c5-a89a-38d664068d7d" containerID="40c09a27e2e396931ae61135b1080d24377a4b23f6174a15a584a20e93ecae9e" exitCode=0 Oct 30 04:28:47.749020 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:47.748671 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" event={"ID":"6118fafe-5cb0-41c5-a89a-38d664068d7d","Type":"ContainerDied","Data":"40c09a27e2e396931ae61135b1080d24377a4b23f6174a15a584a20e93ecae9e"} Oct 30 04:28:48.888208 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:48.888181 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:49.023769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.023684 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\") pod \"6118fafe-5cb0-41c5-a89a-38d664068d7d\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " Oct 30 04:28:49.023769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.023753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\") pod \"6118fafe-5cb0-41c5-a89a-38d664068d7d\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " Oct 30 04:28:49.023983 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.023783 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\" (UniqueName: \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\") pod \"6118fafe-5cb0-41c5-a89a-38d664068d7d\" (UID: \"6118fafe-5cb0-41c5-a89a-38d664068d7d\") " Oct 30 04:28:49.024151 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.024125 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca" pod "6118fafe-5cb0-41c5-a89a-38d664068d7d" (UID: "6118fafe-5cb0-41c5-a89a-38d664068d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:49.025975 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.025944 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca" pod "6118fafe-5cb0-41c5-a89a-38d664068d7d" (UID: "6118fafe-5cb0-41c5-a89a-38d664068d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:49.026090 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.025980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn" pod "6118fafe-5cb0-41c5-a89a-38d664068d7d" (UID: "6118fafe-5cb0-41c5-a89a-38d664068d7d"). InnerVolumeSpecName "kube-api-access-hklrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:49.124876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.124837 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:49.124876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.124872 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6118fafe-5cb0-41c5-a89a-38d664068d7d-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:49.124876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.124882 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hklrn\" (UniqueName: \"kubernetes.io/projected/6118fafe-5cb0-41c5-a89a-38d664068d7d-kube-api-access-hklrn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:49.759383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.759346 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" Oct 30 04:28:49.759383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.759367 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg" event={"ID":"6118fafe-5cb0-41c5-a89a-38d664068d7d","Type":"ContainerDied","Data":"546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339"} Oct 30 04:28:49.759655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:49.759413 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546e3d37f96f8572c16679e98bd0e967300c9ed3cc07d4a7a4485fcb69565339" Oct 30 04:28:55.561668 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.561631 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc"] Oct 30 04:28:55.562088 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.562059 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6118fafe-5cb0-41c5-a89a-38d664068d7d" containerName="revoke-keys" Oct 30 04:28:55.562088 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.562071 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6118fafe-5cb0-41c5-a89a-38d664068d7d" containerName="revoke-keys" Oct 30 04:28:55.562194 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.562148 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6118fafe-5cb0-41c5-a89a-38d664068d7d" containerName="revoke-keys" Oct 30 04:28:55.566585 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.566565 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.584684 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.584653 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc"] Oct 30 04:28:55.586393 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.586365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.586511 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.586476 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgl6\" (UniqueName: \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.586592 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.586510 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.595561 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.595513 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:28:55.595561 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.595527 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:28:55.595756 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.595649 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:28:55.595846 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.595826 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:28:55.687135 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.687096 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.687334 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.687212 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.687334 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.687291 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgl6\" (UniqueName: \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.687857 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.687829 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.689573 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.689542 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.707011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.706980 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgl6\" (UniqueName: \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\") pod \"maas-api-revoke-sub-abc831e5-x5zdc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:55.879680 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:55.879594 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:56.179950 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:56.179919 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc"] Oct 30 04:28:56.183381 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:28:56.183353 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edefc74_23f3_4ede_a5fd_74837106d9cc.slice/crio-7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5 WatchSource:0}: Error finding container 7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5: Status 404 returned error can't find the container with id 7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5 Oct 30 04:28:56.793927 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:56.793829 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" event={"ID":"7edefc74-23f3-4ede-a5fd-74837106d9cc","Type":"ContainerStarted","Data":"83b0adf04ca17c1b6b6b49ee264b0bbdf2436e38a9e8749f7fbe083765af2c29"} Oct 30 04:28:56.793927 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:56.793887 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" event={"ID":"7edefc74-23f3-4ede-a5fd-74837106d9cc","Type":"ContainerStarted","Data":"7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5"} Oct 30 04:28:56.834014 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:56.833948 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" podStartSLOduration=1.833933734 podStartE2EDuration="1.833933734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:56.832324804 +0000 UTC m=+5164.817444695" watchObservedRunningTime="2026-10-30 04:28:56.833933734 +0000 UTC m=+5164.819053621" Oct 30 04:28:57.801327 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:57.801289 4560 generic.go:356] "Generic (PLEG): container finished" podID="7edefc74-23f3-4ede-a5fd-74837106d9cc" containerID="83b0adf04ca17c1b6b6b49ee264b0bbdf2436e38a9e8749f7fbe083765af2c29" exitCode=0 Oct 30 04:28:57.802068 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:57.801374 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" event={"ID":"7edefc74-23f3-4ede-a5fd-74837106d9cc","Type":"ContainerDied","Data":"83b0adf04ca17c1b6b6b49ee264b0bbdf2436e38a9e8749f7fbe083765af2c29"} Oct 30 04:28:58.942040 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:58.942009 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:28:59.018653 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.018612 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\") pod \"7edefc74-23f3-4ede-a5fd-74837106d9cc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " Oct 30 04:28:59.018833 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.018668 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\") pod \"7edefc74-23f3-4ede-a5fd-74837106d9cc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " Oct 30 04:28:59.018884 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.018828 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\" (UniqueName: \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\") pod \"7edefc74-23f3-4ede-a5fd-74837106d9cc\" (UID: \"7edefc74-23f3-4ede-a5fd-74837106d9cc\") " Oct 30 04:28:59.019111 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.019088 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca" pod "7edefc74-23f3-4ede-a5fd-74837106d9cc" (UID: "7edefc74-23f3-4ede-a5fd-74837106d9cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:59.020784 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.020756 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca" pod "7edefc74-23f3-4ede-a5fd-74837106d9cc" (UID: "7edefc74-23f3-4ede-a5fd-74837106d9cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:59.021262 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.021241 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6" pod "7edefc74-23f3-4ede-a5fd-74837106d9cc" (UID: "7edefc74-23f3-4ede-a5fd-74837106d9cc"). InnerVolumeSpecName "kube-api-access-txgl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:59.120085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.119985 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgl6\" (UniqueName: \"kubernetes.io/projected/7edefc74-23f3-4ede-a5fd-74837106d9cc-kube-api-access-txgl6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:59.120085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.120030 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:59.120085 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.120040 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edefc74-23f3-4ede-a5fd-74837106d9cc-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:28:59.813556 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.813502 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" event={"ID":"7edefc74-23f3-4ede-a5fd-74837106d9cc","Type":"ContainerDied","Data":"7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5"} Oct 30 04:28:59.813738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.813600 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7951b76c8e09f3478f5f833a739a1123ed624c5de4baf33cd51ac468eeae77c5" Oct 30 04:28:59.813738 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:28:59.813509 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc" Oct 30 04:29:11.749593 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:29:11.749494 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:29:19.549461 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.549421 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj"] Oct 30 04:29:19.549951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.549894 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7edefc74-23f3-4ede-a5fd-74837106d9cc" containerName="revoke-keys" Oct 30 04:29:19.549951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.549911 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edefc74-23f3-4ede-a5fd-74837106d9cc" containerName="revoke-keys" Oct 30 04:29:19.550032 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.550001 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edefc74-23f3-4ede-a5fd-74837106d9cc" containerName="revoke-keys" Oct 30 04:29:19.553092 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.553074 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.569301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.569268 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:29:19.571575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.571545 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj"] Oct 30 04:29:19.581968 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.581941 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:29:19.581968 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.581942 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:29:19.582140 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.582038 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:29:19.656245 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.656201 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kj2\" (UniqueName: \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.656429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.656254 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.656429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.656396 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.757742 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.757693 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.757950 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.757805 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kj2\" (UniqueName: \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.757950 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.757854 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.758393 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.758372 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.760213 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.760186 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.778186 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.778153 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kj2\" (UniqueName: \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\") pod \"maas-api-revoke-sub-a4b0c39b-lz4nj\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:19.865724 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:19.865632 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:20.164844 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:20.164813 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj"] Oct 30 04:29:20.165825 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:29:20.165794 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6109b196_ff73_49ff_9ed9_e1e92eb87f69.slice/crio-24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3 WatchSource:0}: Error finding container 24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3: Status 404 returned error can't find the container with id 24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3 Oct 30 04:29:20.954016 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:20.953982 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" event={"ID":"6109b196-ff73-49ff-9ed9-e1e92eb87f69","Type":"ContainerStarted","Data":"05dffd314fa4f9a5b918aee7a6b7b221f62073264b99ffc9cc45faa3ba8e3b3f"} Oct 30 04:29:20.954471 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:20.954043 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" event={"ID":"6109b196-ff73-49ff-9ed9-e1e92eb87f69","Type":"ContainerStarted","Data":"24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3"} Oct 30 04:29:20.996736 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:20.996677 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" podStartSLOduration=1.996662433 podStartE2EDuration="1.996662433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:20.995024626 +0000 UTC m=+5188.980144515" watchObservedRunningTime="2026-10-30 04:29:20.996662433 +0000 UTC m=+5188.981782322" Oct 30 04:29:21.960157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:21.960119 4560 generic.go:356] "Generic (PLEG): container finished" podID="6109b196-ff73-49ff-9ed9-e1e92eb87f69" containerID="05dffd314fa4f9a5b918aee7a6b7b221f62073264b99ffc9cc45faa3ba8e3b3f" exitCode=0 Oct 30 04:29:21.960657 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:21.960202 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" event={"ID":"6109b196-ff73-49ff-9ed9-e1e92eb87f69","Type":"ContainerDied","Data":"05dffd314fa4f9a5b918aee7a6b7b221f62073264b99ffc9cc45faa3ba8e3b3f"} Oct 30 04:29:23.100793 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.100769 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:23.195502 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.195466 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\" (UniqueName: \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\") pod \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " Oct 30 04:29:23.195721 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.195513 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\") pod \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " Oct 30 04:29:23.195721 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.195618 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\") pod \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\" (UID: \"6109b196-ff73-49ff-9ed9-e1e92eb87f69\") " Oct 30 04:29:23.196018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.195987 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca" pod "6109b196-ff73-49ff-9ed9-e1e92eb87f69" (UID: "6109b196-ff73-49ff-9ed9-e1e92eb87f69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:23.197760 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.197735 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca" pod "6109b196-ff73-49ff-9ed9-e1e92eb87f69" (UID: "6109b196-ff73-49ff-9ed9-e1e92eb87f69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:23.197876 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.197841 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2" pod "6109b196-ff73-49ff-9ed9-e1e92eb87f69" (UID: "6109b196-ff73-49ff-9ed9-e1e92eb87f69"). InnerVolumeSpecName "kube-api-access-74kj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:23.297274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.297183 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74kj2\" (UniqueName: \"kubernetes.io/projected/6109b196-ff73-49ff-9ed9-e1e92eb87f69-kube-api-access-74kj2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:23.297274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.297219 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:23.297274 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.297233 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6109b196-ff73-49ff-9ed9-e1e92eb87f69-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:23.971359 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.971329 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" Oct 30 04:29:23.971563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.971321 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj" event={"ID":"6109b196-ff73-49ff-9ed9-e1e92eb87f69","Type":"ContainerDied","Data":"24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3"} Oct 30 04:29:23.971563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:23.971490 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24196b7c72f1463fc3c6e5d1916365f7b8d20b195b3b77934278e42c333136d3" Oct 30 04:29:41.749107 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:29:41.749067 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:29:50.151828 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.151795 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb"] Oct 30 04:29:50.152235 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.152187 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6109b196-ff73-49ff-9ed9-e1e92eb87f69" containerName="revoke-keys" Oct 30 04:29:50.152235 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.152197 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6109b196-ff73-49ff-9ed9-e1e92eb87f69" containerName="revoke-keys" Oct 30 04:29:50.152316 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.152267 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6109b196-ff73-49ff-9ed9-e1e92eb87f69" containerName="revoke-keys" Oct 30 04:29:50.155452 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.155429 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.172069 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.172043 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:29:50.185172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.185144 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:29:50.185172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.185164 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:29:50.185405 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.185221 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:29:50.187064 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.187024 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb"] Oct 30 04:29:50.313410 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.313365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.313662 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.313439 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lzl\" (UniqueName: \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.313662 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.313503 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.414966 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.414874 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.415152 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.414979 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.415152 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.415022 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lzl\" (UniqueName: \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.415686 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.415659 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.417412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.417394 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.435114 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.435084 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lzl\" (UniqueName: \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\") pod \"maas-api-revoke-sub-07d50594-pprcb\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.468196 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.468166 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:50.766533 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:50.766370 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb"] Oct 30 04:29:50.769037 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:29:50.769008 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d73e27_fccd_4542_90f4_77f57f8afc11.slice/crio-9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b WatchSource:0}: Error finding container 9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b: Status 404 returned error can't find the container with id 9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b Oct 30 04:29:51.121623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:51.121504 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" event={"ID":"28d73e27-fccd-4542-90f4-77f57f8afc11","Type":"ContainerStarted","Data":"f905179f2d7d640b7416823f3e583981edd5e7e8b7474f4e697727a9f60c858e"} Oct 30 04:29:51.121623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:51.121592 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" event={"ID":"28d73e27-fccd-4542-90f4-77f57f8afc11","Type":"ContainerStarted","Data":"9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b"} Oct 30 04:29:51.163901 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:51.163842 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" podStartSLOduration=1.163828785 podStartE2EDuration="1.163828785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:51.162792745 +0000 UTC m=+5219.147912624" watchObservedRunningTime="2026-10-30 04:29:51.163828785 +0000 UTC m=+5219.148948686" Oct 30 04:29:52.127365 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:52.127331 4560 generic.go:356] "Generic (PLEG): container finished" podID="28d73e27-fccd-4542-90f4-77f57f8afc11" containerID="f905179f2d7d640b7416823f3e583981edd5e7e8b7474f4e697727a9f60c858e" exitCode=0 Oct 30 04:29:52.127572 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:52.127406 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" event={"ID":"28d73e27-fccd-4542-90f4-77f57f8afc11","Type":"ContainerDied","Data":"f905179f2d7d640b7416823f3e583981edd5e7e8b7474f4e697727a9f60c858e"} Oct 30 04:29:53.269530 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.269491 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:53.446089 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.446003 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\") pod \"28d73e27-fccd-4542-90f4-77f57f8afc11\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " Oct 30 04:29:53.446268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.446144 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\") pod \"28d73e27-fccd-4542-90f4-77f57f8afc11\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " Oct 30 04:29:53.446268 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.446215 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\" (UniqueName: \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\") pod \"28d73e27-fccd-4542-90f4-77f57f8afc11\" (UID: \"28d73e27-fccd-4542-90f4-77f57f8afc11\") " Oct 30 04:29:53.446614 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.446581 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca" pod "28d73e27-fccd-4542-90f4-77f57f8afc11" (UID: "28d73e27-fccd-4542-90f4-77f57f8afc11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:53.448261 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.448229 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca" pod "28d73e27-fccd-4542-90f4-77f57f8afc11" (UID: "28d73e27-fccd-4542-90f4-77f57f8afc11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:53.448261 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.448251 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl" pod "28d73e27-fccd-4542-90f4-77f57f8afc11" (UID: "28d73e27-fccd-4542-90f4-77f57f8afc11"). InnerVolumeSpecName "kube-api-access-72lzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:53.547994 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.547950 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72lzl\" (UniqueName: \"kubernetes.io/projected/28d73e27-fccd-4542-90f4-77f57f8afc11-kube-api-access-72lzl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.547994 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.547988 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.547994 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:53.547997 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d73e27-fccd-4542-90f4-77f57f8afc11-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:29:54.138941 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:54.138910 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" Oct 30 04:29:54.138941 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:54.138925 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d50594-pprcb" event={"ID":"28d73e27-fccd-4542-90f4-77f57f8afc11","Type":"ContainerDied","Data":"9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b"} Oct 30 04:29:54.139160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:29:54.138992 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9237a0dc9779bb61cd566de47e41511e5d23bd5d85cf2a7ef1b62a98c1d4ef6b" Oct 30 04:30:00.135431 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.135396 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:00.135923 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.135907 4560 cpu_manager.go:414] "containerMap: removing container" podUID="28d73e27-fccd-4542-90f4-77f57f8afc11" containerName="revoke-keys" Oct 30 04:30:00.135969 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.135924 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d73e27-fccd-4542-90f4-77f57f8afc11" containerName="revoke-keys" Oct 30 04:30:00.136017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.136007 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d73e27-fccd-4542-90f4-77f57f8afc11" containerName="revoke-keys" Oct 30 04:30:00.139567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.139546 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.158033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.157999 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:00.168810 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.168756 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:30:00.169367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.169091 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:30:00.169367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.169194 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:30:00.211380 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.211330 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9v2\" (UniqueName: \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.211573 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.211531 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.312834 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.312790 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.313017 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.312887 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9v2\" (UniqueName: \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.313560 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.313505 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.333120 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.333085 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9v2\" (UniqueName: \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\") pod \"maas-api-key-cleanup-29888910-jth6f\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.452618 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.452588 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:00.755703 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:00.755673 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:00.757398 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:30:00.757369 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac167859_e38e_49d7_8586_f8abd44f614b.slice/crio-dd896e9fc2a47b70345ae26c0a7b05c0f9e3970b0c0c27045d1594bf51980fe2 WatchSource:0}: Error finding container dd896e9fc2a47b70345ae26c0a7b05c0f9e3970b0c0c27045d1594bf51980fe2: Status 404 returned error can't find the container with id dd896e9fc2a47b70345ae26c0a7b05c0f9e3970b0c0c27045d1594bf51980fe2 Oct 30 04:30:01.183543 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:01.183480 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerStarted","Data":"cfcff34a08f8ce581b1be4711c85348dd137ccc4c72cf96d4bfc58a0da35b557"} Oct 30 04:30:01.184052 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:01.183568 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerStarted","Data":"dd896e9fc2a47b70345ae26c0a7b05c0f9e3970b0c0c27045d1594bf51980fe2"} Oct 30 04:30:01.233888 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:01.233831 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" podStartSLOduration=1.233813951 podStartE2EDuration="1.233813951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:01.233790062 +0000 UTC m=+5229.218909950" watchObservedRunningTime="2026-10-30 04:30:01.233813951 +0000 UTC m=+5229.218933840" Oct 30 04:30:02.190218 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:02.190180 4560 generic.go:356] "Generic (PLEG): container finished" podID="ac167859-e38e-49d7-8586-f8abd44f614b" containerID="cfcff34a08f8ce581b1be4711c85348dd137ccc4c72cf96d4bfc58a0da35b557" exitCode=60 Oct 30 04:30:02.190218 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:02.190220 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerDied","Data":"cfcff34a08f8ce581b1be4711c85348dd137ccc4c72cf96d4bfc58a0da35b557"} Oct 30 04:30:02.190794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:02.190572 4560 scope.go:122] "RemoveContainer" containerID="cfcff34a08f8ce581b1be4711c85348dd137ccc4c72cf96d4bfc58a0da35b557" Oct 30 04:30:03.197445 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:03.197414 4560 generic.go:356] "Generic (PLEG): container finished" podID="ac167859-e38e-49d7-8586-f8abd44f614b" containerID="32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26" exitCode=60 Oct 30 04:30:03.197916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:03.197475 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerDied","Data":"32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26"} Oct 30 04:30:03.197916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:03.197540 4560 scope.go:122] "RemoveContainer" containerID="cfcff34a08f8ce581b1be4711c85348dd137ccc4c72cf96d4bfc58a0da35b557" Oct 30 04:30:03.197916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:03.197792 4560 scope.go:122] "RemoveContainer" containerID="32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26" Oct 30 04:30:03.198087 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:30:03.198056 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-jth6f_odh-ai-gateway-infra(ac167859-e38e-49d7-8586-f8abd44f614b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" Oct 30 04:30:04.203835 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:04.203798 4560 scope.go:122] "RemoveContainer" containerID="32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26" Oct 30 04:30:04.204224 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:30:04.204036 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-jth6f_odh-ai-gateway-infra(ac167859-e38e-49d7-8586-f8abd44f614b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" Oct 30 04:30:11.748606 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:30:11.748563 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:30:15.145527 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:15.145485 4560 scope.go:122] "RemoveContainer" containerID="32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26" Oct 30 04:30:16.304935 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:16.303974 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerStarted","Data":"93c0243ef2ab801961186752792af3f91bd31d1f77001ba21a4c8ee8e21ea802"} Oct 30 04:30:17.310818 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:17.310784 4560 generic.go:356] "Generic (PLEG): container finished" podID="ac167859-e38e-49d7-8586-f8abd44f614b" containerID="93c0243ef2ab801961186752792af3f91bd31d1f77001ba21a4c8ee8e21ea802" exitCode=60 Oct 30 04:30:17.311255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:17.310853 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerDied","Data":"93c0243ef2ab801961186752792af3f91bd31d1f77001ba21a4c8ee8e21ea802"} Oct 30 04:30:17.311255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:17.310914 4560 scope.go:122] "RemoveContainer" containerID="32bdc2a7beca2f6c866e0c5c405590cd26afae3a2a2bc79edc3de28b7e9bcc26" Oct 30 04:30:17.311255 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:17.311251 4560 scope.go:122] "RemoveContainer" containerID="93c0243ef2ab801961186752792af3f91bd31d1f77001ba21a4c8ee8e21ea802" Oct 30 04:30:17.311583 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:30:17.311557 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-jth6f_odh-ai-gateway-infra(ac167859-e38e-49d7-8586-f8abd44f614b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" Oct 30 04:30:17.358569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:17.358530 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:18.553818 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.553784 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:18.648281 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.648201 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\" (UniqueName: \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\") pod \"ac167859-e38e-49d7-8586-f8abd44f614b\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " Oct 30 04:30:18.648281 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.648268 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\") pod \"ac167859-e38e-49d7-8586-f8abd44f614b\" (UID: \"ac167859-e38e-49d7-8586-f8abd44f614b\") " Oct 30 04:30:18.653540 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.648705 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca" pod "ac167859-e38e-49d7-8586-f8abd44f614b" (UID: "ac167859-e38e-49d7-8586-f8abd44f614b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:18.669027 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.668978 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2" pod "ac167859-e38e-49d7-8586-f8abd44f614b" (UID: "ac167859-e38e-49d7-8586-f8abd44f614b"). InnerVolumeSpecName "kube-api-access-df9v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:18.753292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.753213 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9v2\" (UniqueName: \"kubernetes.io/projected/ac167859-e38e-49d7-8586-f8abd44f614b-kube-api-access-df9v2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:18.753292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:18.753254 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac167859-e38e-49d7-8586-f8abd44f614b-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:19.327553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:19.327461 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" event={"ID":"ac167859-e38e-49d7-8586-f8abd44f614b","Type":"ContainerDied","Data":"dd896e9fc2a47b70345ae26c0a7b05c0f9e3970b0c0c27045d1594bf51980fe2"} Oct 30 04:30:19.327770 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:19.327480 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f" Oct 30 04:30:19.328067 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:19.327563 4560 scope.go:122] "RemoveContainer" containerID="93c0243ef2ab801961186752792af3f91bd31d1f77001ba21a4c8ee8e21ea802" Oct 30 04:30:19.413542 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:19.411619 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:19.415567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:19.414560 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-jth6f"] Oct 30 04:30:20.151596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:20.151566 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" path="/var/lib/kubelet/pods/ac167859-e38e-49d7-8586-f8abd44f614b/volumes" Oct 30 04:30:29.484131 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484093 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85"] Oct 30 04:30:29.484764 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484740 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.484848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484766 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.484848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484823 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.484848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484832 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.485015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484950 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.485015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.484972 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:29.489653 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.489631 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.506804 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.506777 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85"] Oct 30 04:30:29.517395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.517372 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:30:29.517501 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.517462 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:30:29.568452 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.568426 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:30:29.568631 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.568503 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:30:29.653616 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.653582 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.653826 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.653623 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppxr\" (UniqueName: \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.653826 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.653809 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.754855 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.754759 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.755041 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.754859 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.755041 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.754892 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppxr\" (UniqueName: \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.755471 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.755439 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.757241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.757221 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.774763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.774737 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppxr\" (UniqueName: \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\") pod \"maas-api-revoke-sub-7c5d2b10-97b85\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:29.802660 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:29.802636 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:30.100037 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:30.100009 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85"] Oct 30 04:30:30.101924 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:30:30.101894 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ae5600_04aa_4e60_a870_f4fb80547ecc.slice/crio-4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015 WatchSource:0}: Error finding container 4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015: Status 404 returned error can't find the container with id 4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015 Oct 30 04:30:30.407292 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:30.407254 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" event={"ID":"e9ae5600-04aa-4e60-a870-f4fb80547ecc","Type":"ContainerStarted","Data":"82f80abc77c2ce53c5b474cb734b5902decd2b4f3d96e81a3b86f44556bdcb84"} Oct 30 04:30:30.407431 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:30.407309 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" event={"ID":"e9ae5600-04aa-4e60-a870-f4fb80547ecc","Type":"ContainerStarted","Data":"4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015"} Oct 30 04:30:30.448364 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:30.448316 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" podStartSLOduration=1.448301201 podStartE2EDuration="1.448301201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:30.446297069 +0000 UTC m=+5258.431416958" watchObservedRunningTime="2026-10-30 04:30:30.448301201 +0000 UTC m=+5258.433421090" Oct 30 04:30:31.413972 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:31.413937 4560 generic.go:356] "Generic (PLEG): container finished" podID="e9ae5600-04aa-4e60-a870-f4fb80547ecc" containerID="82f80abc77c2ce53c5b474cb734b5902decd2b4f3d96e81a3b86f44556bdcb84" exitCode=0 Oct 30 04:30:31.414418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:31.414028 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" event={"ID":"e9ae5600-04aa-4e60-a870-f4fb80547ecc","Type":"ContainerDied","Data":"82f80abc77c2ce53c5b474cb734b5902decd2b4f3d96e81a3b86f44556bdcb84"} Oct 30 04:30:32.555620 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.555592 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:32.685757 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.685676 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\") pod \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " Oct 30 04:30:32.685921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.685814 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\" (UniqueName: \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\") pod \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " Oct 30 04:30:32.685921 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.685914 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\") pod \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\" (UID: \"e9ae5600-04aa-4e60-a870-f4fb80547ecc\") " Oct 30 04:30:32.686130 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.686107 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca" pod "e9ae5600-04aa-4e60-a870-f4fb80547ecc" (UID: "e9ae5600-04aa-4e60-a870-f4fb80547ecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:32.686250 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.686229 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:32.687959 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.687925 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr" pod "e9ae5600-04aa-4e60-a870-f4fb80547ecc" (UID: "e9ae5600-04aa-4e60-a870-f4fb80547ecc"). InnerVolumeSpecName "kube-api-access-4ppxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:32.688065 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.687978 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca" pod "e9ae5600-04aa-4e60-a870-f4fb80547ecc" (UID: "e9ae5600-04aa-4e60-a870-f4fb80547ecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:30:32.787785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.787742 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ppxr\" (UniqueName: \"kubernetes.io/projected/e9ae5600-04aa-4e60-a870-f4fb80547ecc-kube-api-access-4ppxr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:32.787785 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:32.787778 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae5600-04aa-4e60-a870-f4fb80547ecc-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:33.425473 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:33.425435 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" event={"ID":"e9ae5600-04aa-4e60-a870-f4fb80547ecc","Type":"ContainerDied","Data":"4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015"} Oct 30 04:30:33.425473 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:33.425465 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d2b10-97b85" Oct 30 04:30:33.425732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:33.425494 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a204b642072490125bda26dc4e2d3dbd60a13e4512e86c5dce3e4f804c38015" Oct 30 04:30:39.780914 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.780826 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7"] Oct 30 04:30:39.781358 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781339 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e9ae5600-04aa-4e60-a870-f4fb80547ecc" containerName="revoke-keys" Oct 30 04:30:39.781408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781359 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ae5600-04aa-4e60-a870-f4fb80547ecc" containerName="revoke-keys" Oct 30 04:30:39.781408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781379 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:39.781408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781385 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:39.781504 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781467 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ae5600-04aa-4e60-a870-f4fb80547ecc" containerName="revoke-keys" Oct 30 04:30:39.781504 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.781477 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac167859-e38e-49d7-8586-f8abd44f614b" containerName="cleanup" Oct 30 04:30:39.784506 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.784490 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.802619 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.802569 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7"] Oct 30 04:30:39.812635 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.812606 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:30:39.812827 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.812623 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:30:39.812892 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.812684 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:30:39.812892 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.812861 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:30:39.858314 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.858280 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.858490 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.858325 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.858490 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.858445 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47pc\" (UniqueName: \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.959266 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.959227 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.959440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.959278 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.959440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.959344 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47pc\" (UniqueName: \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.959978 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.959956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.961863 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.961834 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:39.979407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:39.979370 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47pc\" (UniqueName: \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\") pod \"maas-api-revoke-sub-a840ec60-z4kz7\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:40.101660 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:40.101574 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:40.399166 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:40.399140 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7"] Oct 30 04:30:40.400966 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:30:40.400934 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd22dcc_b40d_4eed_b711_17f37259bd85.slice/crio-8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4 WatchSource:0}: Error finding container 8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4: Status 404 returned error can't find the container with id 8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4 Oct 30 04:30:40.458989 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:40.458948 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" event={"ID":"9bd22dcc-b40d-4eed-b711-17f37259bd85","Type":"ContainerStarted","Data":"8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4"} Oct 30 04:30:41.467882 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:41.467840 4560 generic.go:356] "Generic (PLEG): container finished" podID="9bd22dcc-b40d-4eed-b711-17f37259bd85" containerID="797d2871b7a220327f459fe9766ce20468c30614af3b7d5f64305aaf8109c78c" exitCode=0 Oct 30 04:30:41.468308 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:41.467926 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" event={"ID":"9bd22dcc-b40d-4eed-b711-17f37259bd85","Type":"ContainerDied","Data":"797d2871b7a220327f459fe9766ce20468c30614af3b7d5f64305aaf8109c78c"} Oct 30 04:30:41.757378 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:30:41.757337 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:30:42.838946 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.838922 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:42.932552 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.932181 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\" (UniqueName: \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\") pod \"9bd22dcc-b40d-4eed-b711-17f37259bd85\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " Oct 30 04:30:42.932552 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.932287 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\") pod \"9bd22dcc-b40d-4eed-b711-17f37259bd85\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " Oct 30 04:30:42.932552 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.932330 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\") pod \"9bd22dcc-b40d-4eed-b711-17f37259bd85\" (UID: \"9bd22dcc-b40d-4eed-b711-17f37259bd85\") " Oct 30 04:30:42.933032 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.933001 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca" pod "9bd22dcc-b40d-4eed-b711-17f37259bd85" (UID: "9bd22dcc-b40d-4eed-b711-17f37259bd85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:42.934866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.934834 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc" pod "9bd22dcc-b40d-4eed-b711-17f37259bd85" (UID: "9bd22dcc-b40d-4eed-b711-17f37259bd85"). InnerVolumeSpecName "kube-api-access-b47pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:42.935136 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:42.935097 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca" pod "9bd22dcc-b40d-4eed-b711-17f37259bd85" (UID: "9bd22dcc-b40d-4eed-b711-17f37259bd85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:30:43.033692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.033636 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47pc\" (UniqueName: \"kubernetes.io/projected/9bd22dcc-b40d-4eed-b711-17f37259bd85-kube-api-access-b47pc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:43.033692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.033685 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:43.033692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.033695 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd22dcc-b40d-4eed-b711-17f37259bd85-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:30:43.520390 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.520252 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" Oct 30 04:30:43.520390 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.520293 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a840ec60-z4kz7" event={"ID":"9bd22dcc-b40d-4eed-b711-17f37259bd85","Type":"ContainerDied","Data":"8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4"} Oct 30 04:30:43.520390 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:30:43.520340 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aef1f5b4d83dfb9f85b2b214afe68805831d208a3e7762bece490e394648aa4" Oct 30 04:31:11.748897 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:31:11.748862 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:31:12.771026 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.770967 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj"] Oct 30 04:31:12.771634 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.771613 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9bd22dcc-b40d-4eed-b711-17f37259bd85" containerName="revoke-keys" Oct 30 04:31:12.771730 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.771637 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd22dcc-b40d-4eed-b711-17f37259bd85" containerName="revoke-keys" Oct 30 04:31:12.771793 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.771782 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd22dcc-b40d-4eed-b711-17f37259bd85" containerName="revoke-keys" Oct 30 04:31:12.774843 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.774821 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.798569 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.798544 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:31:12.798678 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.798637 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:31:12.807054 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.807027 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:31:12.807217 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.807168 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:31:12.810015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.809987 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj"] Oct 30 04:31:12.830202 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.830170 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wwf\" (UniqueName: \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.830322 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.830236 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.830322 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.830307 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.931701 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.931666 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.931875 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.931705 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.931875 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.931823 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wwf\" (UniqueName: \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.932395 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.932364 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.934227 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.934201 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:12.951208 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:12.951184 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wwf\" (UniqueName: \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\") pod \"maas-api-revoke-sub-2860ee91-sbgwj\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:13.088027 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:13.087934 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:13.386819 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:13.386788 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj"] Oct 30 04:31:13.387909 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:31:13.387881 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc283196a_fe8c_4367_8b9d_d1a23463adf8.slice/crio-51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e WatchSource:0}: Error finding container 51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e: Status 404 returned error can't find the container with id 51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e Oct 30 04:31:13.749037 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:13.748998 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" event={"ID":"c283196a-fe8c-4367-8b9d-d1a23463adf8","Type":"ContainerStarted","Data":"f5fac95c781e14431f2111d79b495cb00f7b5052560f1de83c20ddb1f086977d"} Oct 30 04:31:13.749257 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:13.749055 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" event={"ID":"c283196a-fe8c-4367-8b9d-d1a23463adf8","Type":"ContainerStarted","Data":"51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e"} Oct 30 04:31:13.789570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:13.789507 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" podStartSLOduration=1.789492845 podStartE2EDuration="1.789492845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:13.787802554 +0000 UTC m=+5301.772922444" watchObservedRunningTime="2026-10-30 04:31:13.789492845 +0000 UTC m=+5301.774612733" Oct 30 04:31:14.756208 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:14.756171 4560 generic.go:356] "Generic (PLEG): container finished" podID="c283196a-fe8c-4367-8b9d-d1a23463adf8" containerID="f5fac95c781e14431f2111d79b495cb00f7b5052560f1de83c20ddb1f086977d" exitCode=0 Oct 30 04:31:14.756410 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:14.756258 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" event={"ID":"c283196a-fe8c-4367-8b9d-d1a23463adf8","Type":"ContainerDied","Data":"f5fac95c781e14431f2111d79b495cb00f7b5052560f1de83c20ddb1f086977d"} Oct 30 04:31:15.889788 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.889763 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:15.963713 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.963683 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\") pod \"c283196a-fe8c-4367-8b9d-d1a23463adf8\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " Oct 30 04:31:15.963856 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.963735 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\") pod \"c283196a-fe8c-4367-8b9d-d1a23463adf8\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " Oct 30 04:31:15.963856 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.963771 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\" (UniqueName: \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\") pod \"c283196a-fe8c-4367-8b9d-d1a23463adf8\" (UID: \"c283196a-fe8c-4367-8b9d-d1a23463adf8\") " Oct 30 04:31:15.964299 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.964265 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca" pod "c283196a-fe8c-4367-8b9d-d1a23463adf8" (UID: "c283196a-fe8c-4367-8b9d-d1a23463adf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:15.967968 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.967942 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca" pod "c283196a-fe8c-4367-8b9d-d1a23463adf8" (UID: "c283196a-fe8c-4367-8b9d-d1a23463adf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:15.968073 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:15.968010 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf" pod "c283196a-fe8c-4367-8b9d-d1a23463adf8" (UID: "c283196a-fe8c-4367-8b9d-d1a23463adf8"). InnerVolumeSpecName "kube-api-access-m4wwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:16.064916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.064831 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:16.064916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.064864 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c283196a-fe8c-4367-8b9d-d1a23463adf8-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:16.064916 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.064874 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wwf\" (UniqueName: \"kubernetes.io/projected/c283196a-fe8c-4367-8b9d-d1a23463adf8-kube-api-access-m4wwf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:16.766445 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.766412 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" Oct 30 04:31:16.766691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.766452 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2860ee91-sbgwj" event={"ID":"c283196a-fe8c-4367-8b9d-d1a23463adf8","Type":"ContainerDied","Data":"51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e"} Oct 30 04:31:16.766691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:16.766504 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51dcd7ceea0fbdd028645f2064ace07080e1d421cc30f346801744e4a754a62e" Oct 30 04:31:23.138243 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.138203 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274"] Oct 30 04:31:23.138686 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.138669 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c283196a-fe8c-4367-8b9d-d1a23463adf8" containerName="revoke-keys" Oct 30 04:31:23.138735 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.138687 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c283196a-fe8c-4367-8b9d-d1a23463adf8" containerName="revoke-keys" Oct 30 04:31:23.138783 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.138773 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c283196a-fe8c-4367-8b9d-d1a23463adf8" containerName="revoke-keys" Oct 30 04:31:23.141955 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.141930 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.158896 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.158868 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:31:23.162362 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.162334 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274"] Oct 30 04:31:23.171758 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.171732 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:31:23.171956 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.171765 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:31:23.171956 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.171831 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:31:23.237867 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.237829 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.238087 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.238065 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ngf\" (UniqueName: \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.238281 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.238257 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.339449 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.339399 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.339659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.339487 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.339659 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.339552 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ngf\" (UniqueName: \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.340203 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.340178 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.341890 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.341867 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.359385 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.359354 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ngf\" (UniqueName: \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\") pod \"maas-api-revoke-sub-d658b3e6-6j274\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.455608 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.455574 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:23.765210 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.765182 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274"] Oct 30 04:31:23.766416 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:31:23.766390 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5dddf7_2de3_4cb1_958e_031a144f5d92.slice/crio-78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2 WatchSource:0}: Error finding container 78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2: Status 404 returned error can't find the container with id 78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2 Oct 30 04:31:23.800562 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:23.800530 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" event={"ID":"dd5dddf7-2de3-4cb1-958e-031a144f5d92","Type":"ContainerStarted","Data":"78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2"} Oct 30 04:31:24.807364 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:24.807327 4560 generic.go:356] "Generic (PLEG): container finished" podID="dd5dddf7-2de3-4cb1-958e-031a144f5d92" containerID="70982662bbf4ed0045841f62d22f8ca78ec69b842487fc00a09f6e4b3d8ff294" exitCode=0 Oct 30 04:31:24.807775 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:24.807421 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" event={"ID":"dd5dddf7-2de3-4cb1-958e-031a144f5d92","Type":"ContainerDied","Data":"70982662bbf4ed0045841f62d22f8ca78ec69b842487fc00a09f6e4b3d8ff294"} Oct 30 04:31:25.972412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:25.972384 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:26.066110 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.066001 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\" (UniqueName: \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\") pod \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " Oct 30 04:31:26.066302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.066143 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\") pod \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " Oct 30 04:31:26.066302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.066201 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\") pod \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\" (UID: \"dd5dddf7-2de3-4cb1-958e-031a144f5d92\") " Oct 30 04:31:26.066579 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.066543 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca" pod "dd5dddf7-2de3-4cb1-958e-031a144f5d92" (UID: "dd5dddf7-2de3-4cb1-958e-031a144f5d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:26.068479 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.068443 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf" pod "dd5dddf7-2de3-4cb1-958e-031a144f5d92" (UID: "dd5dddf7-2de3-4cb1-958e-031a144f5d92"). InnerVolumeSpecName "kube-api-access-j9ngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:26.068874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.068846 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca" pod "dd5dddf7-2de3-4cb1-958e-031a144f5d92" (UID: "dd5dddf7-2de3-4cb1-958e-031a144f5d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:26.167105 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.167073 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ngf\" (UniqueName: \"kubernetes.io/projected/dd5dddf7-2de3-4cb1-958e-031a144f5d92-kube-api-access-j9ngf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:26.167105 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.167100 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:26.167326 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.167112 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5dddf7-2de3-4cb1-958e-031a144f5d92-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:26.824549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.823903 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" event={"ID":"dd5dddf7-2de3-4cb1-958e-031a144f5d92","Type":"ContainerDied","Data":"78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2"} Oct 30 04:31:26.824549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.823968 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78498d39f10ba8f098dbb20433716c0b4e867c83c7a97af4e6077ba315db2fa2" Oct 30 04:31:26.824549 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:26.824039 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d658b3e6-6j274" Oct 30 04:31:33.456445 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.456404 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl"] Oct 30 04:31:33.456951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.456862 4560 cpu_manager.go:414] "containerMap: removing container" podUID="dd5dddf7-2de3-4cb1-958e-031a144f5d92" containerName="revoke-keys" Oct 30 04:31:33.456951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.456876 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5dddf7-2de3-4cb1-958e-031a144f5d92" containerName="revoke-keys" Oct 30 04:31:33.456951 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.456939 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5dddf7-2de3-4cb1-958e-031a144f5d92" containerName="revoke-keys" Oct 30 04:31:33.460541 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.460502 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.479313 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.479267 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl"] Oct 30 04:31:33.489791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.489757 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:31:33.489791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.489767 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:31:33.490000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.489835 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:31:33.490000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.489897 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:31:33.573134 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.573094 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x5p\" (UniqueName: \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.573317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.573188 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.573317 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.573283 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.674200 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.674142 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x5p\" (UniqueName: \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.674391 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.674202 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.674391 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.674263 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.674964 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.674932 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.676657 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.676637 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.694848 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.694822 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x5p\" (UniqueName: \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\") pod \"maas-api-revoke-sub-4aeb5da1-gjwtl\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:33.775241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:33.775146 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:34.076918 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:34.076886 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl"] Oct 30 04:31:34.077677 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:31:34.077649 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a04f8c_f9f6_4577_b307_3bfc3b13b8cb.slice/crio-732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0 WatchSource:0}: Error finding container 732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0: Status 404 returned error can't find the container with id 732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0 Oct 30 04:31:34.868389 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:34.868351 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" event={"ID":"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb","Type":"ContainerStarted","Data":"8a44927eb336fc614a2394d66be78980f51f2491211786535f72218bf4a5e51e"} Oct 30 04:31:34.868868 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:34.868411 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" event={"ID":"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb","Type":"ContainerStarted","Data":"732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0"} Oct 30 04:31:34.910149 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:34.910095 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" podStartSLOduration=1.910079793 podStartE2EDuration="1.910079793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:34.907953621 +0000 UTC m=+5322.893073544" watchObservedRunningTime="2026-10-30 04:31:34.910079793 +0000 UTC m=+5322.895199731" Oct 30 04:31:35.875632 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:35.875600 4560 generic.go:356] "Generic (PLEG): container finished" podID="86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" containerID="8a44927eb336fc614a2394d66be78980f51f2491211786535f72218bf4a5e51e" exitCode=0 Oct 30 04:31:35.876029 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:35.875680 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" event={"ID":"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb","Type":"ContainerDied","Data":"8a44927eb336fc614a2394d66be78980f51f2491211786535f72218bf4a5e51e"} Oct 30 04:31:37.022020 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.021990 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:37.111296 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.111248 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\" (UniqueName: \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\") pod \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " Oct 30 04:31:37.111496 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.111344 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\") pod \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " Oct 30 04:31:37.111496 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.111470 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\") pod \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\" (UID: \"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb\") " Oct 30 04:31:37.111739 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.111712 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca" pod "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" (UID: "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:37.111881 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.111862 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:37.113408 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.113386 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca" pod "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" (UID: "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:37.113572 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.113554 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p" pod "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" (UID: "86a04f8c-f9f6-4577-b307-3bfc3b13b8cb"). InnerVolumeSpecName "kube-api-access-24x5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:37.213348 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.213303 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x5p\" (UniqueName: \"kubernetes.io/projected/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-kube-api-access-24x5p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:37.213348 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.213347 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a04f8c-f9f6-4577-b307-3bfc3b13b8cb-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:37.886320 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.886292 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" Oct 30 04:31:37.886502 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.886293 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb5da1-gjwtl" event={"ID":"86a04f8c-f9f6-4577-b307-3bfc3b13b8cb","Type":"ContainerDied","Data":"732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0"} Oct 30 04:31:37.886502 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:37.886420 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732721e47e9c64be9b6b44cadc244222f97f72312216a33b9c9cb9b256ca5ff0" Oct 30 04:31:41.748902 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:31:41.748867 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:31:43.815094 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.815056 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44"] Oct 30 04:31:43.815647 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.815533 4560 cpu_manager.go:414] "containerMap: removing container" podUID="86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" containerName="revoke-keys" Oct 30 04:31:43.815647 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.815548 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" containerName="revoke-keys" Oct 30 04:31:43.815647 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.815612 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a04f8c-f9f6-4577-b307-3bfc3b13b8cb" containerName="revoke-keys" Oct 30 04:31:43.818778 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.818758 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.834709 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.834683 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:31:43.834853 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.834748 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:31:43.847567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.847537 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:31:43.847567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.847563 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:31:43.851486 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.851458 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44"] Oct 30 04:31:43.883048 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.883016 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ts9r\" (UniqueName: \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.883230 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.883076 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.883230 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.883154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.983784 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.983750 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.983959 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.983846 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ts9r\" (UniqueName: \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.983959 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.983896 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.984583 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.984561 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:43.986151 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:43.986132 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:44.004421 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.004393 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ts9r\" (UniqueName: \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\") pod \"maas-api-revoke-sub-ea2dc8fe-jqg44\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:44.132489 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.132402 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:44.434986 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.434952 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44"] Oct 30 04:31:44.436964 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:31:44.436935 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc285c2b5_839e_4395_b801_123c3a7e43b1.slice/crio-f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df WatchSource:0}: Error finding container f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df: Status 404 returned error can't find the container with id f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df Oct 30 04:31:44.926161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.926120 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" event={"ID":"c285c2b5-839e-4395-b801-123c3a7e43b1","Type":"ContainerStarted","Data":"bced17710f806c1acfd0f0c491ac7da52a985124e9e9e2c79de931039a4a35b9"} Oct 30 04:31:44.926603 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.926184 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" event={"ID":"c285c2b5-839e-4395-b801-123c3a7e43b1","Type":"ContainerStarted","Data":"f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df"} Oct 30 04:31:44.967874 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:44.967811 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" podStartSLOduration=1.967797609 podStartE2EDuration="1.967797609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:44.965726438 +0000 UTC m=+5332.950846326" watchObservedRunningTime="2026-10-30 04:31:44.967797609 +0000 UTC m=+5332.952917497" Oct 30 04:31:45.936199 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:45.936163 4560 generic.go:356] "Generic (PLEG): container finished" podID="c285c2b5-839e-4395-b801-123c3a7e43b1" containerID="bced17710f806c1acfd0f0c491ac7da52a985124e9e9e2c79de931039a4a35b9" exitCode=0 Oct 30 04:31:45.936663 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:45.936253 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" event={"ID":"c285c2b5-839e-4395-b801-123c3a7e43b1","Type":"ContainerDied","Data":"bced17710f806c1acfd0f0c491ac7da52a985124e9e9e2c79de931039a4a35b9"} Oct 30 04:31:47.264564 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.264351 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:47.340993 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.340952 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\") pod \"c285c2b5-839e-4395-b801-123c3a7e43b1\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " Oct 30 04:31:47.341265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.341002 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\") pod \"c285c2b5-839e-4395-b801-123c3a7e43b1\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " Oct 30 04:31:47.341265 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.341027 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\" (UniqueName: \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\") pod \"c285c2b5-839e-4395-b801-123c3a7e43b1\" (UID: \"c285c2b5-839e-4395-b801-123c3a7e43b1\") " Oct 30 04:31:47.341687 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.341658 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca" pod "c285c2b5-839e-4395-b801-123c3a7e43b1" (UID: "c285c2b5-839e-4395-b801-123c3a7e43b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:47.343387 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.343353 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r" pod "c285c2b5-839e-4395-b801-123c3a7e43b1" (UID: "c285c2b5-839e-4395-b801-123c3a7e43b1"). InnerVolumeSpecName "kube-api-access-9ts9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:47.343769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.343749 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca" pod "c285c2b5-839e-4395-b801-123c3a7e43b1" (UID: "c285c2b5-839e-4395-b801-123c3a7e43b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:47.442440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.442396 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:47.442440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.442431 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c285c2b5-839e-4395-b801-123c3a7e43b1-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:47.442440 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.442441 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ts9r\" (UniqueName: \"kubernetes.io/projected/c285c2b5-839e-4395-b801-123c3a7e43b1-kube-api-access-9ts9r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:47.946744 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.946707 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" event={"ID":"c285c2b5-839e-4395-b801-123c3a7e43b1","Type":"ContainerDied","Data":"f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df"} Oct 30 04:31:47.946898 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.946766 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e1efe5f99722af8d27c0f9f3a2760f9f32f5ec6dd791bc945d395263c9f3df" Oct 30 04:31:47.946898 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:47.946728 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2dc8fe-jqg44" Oct 30 04:31:54.155909 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.155877 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72"] Oct 30 04:31:54.156324 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.156305 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c285c2b5-839e-4395-b801-123c3a7e43b1" containerName="revoke-keys" Oct 30 04:31:54.156373 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.156327 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c285c2b5-839e-4395-b801-123c3a7e43b1" containerName="revoke-keys" Oct 30 04:31:54.156430 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.156421 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c285c2b5-839e-4395-b801-123c3a7e43b1" containerName="revoke-keys" Oct 30 04:31:54.159847 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.159819 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.176742 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.176715 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:31:54.176742 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.176718 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:31:54.189613 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.189572 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:31:54.189613 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.189602 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:31:54.192287 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.192263 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72"] Oct 30 04:31:54.204912 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.204888 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.205075 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.205046 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klxb\" (UniqueName: \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.205146 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.205115 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.306623 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.306584 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klxb\" (UniqueName: \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.306815 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.306653 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.306815 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.306684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.307346 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.307318 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.309036 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.309005 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.326493 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.326467 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klxb\" (UniqueName: \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\") pod \"maas-api-revoke-sub-f74b8af6-ckc72\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.474176 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.474144 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:54.773186 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.773156 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72"] Oct 30 04:31:54.775560 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:31:54.775529 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4661b0bf_74a1_4221_a09f_7c74ba2efef7.slice/crio-f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078 WatchSource:0}: Error finding container f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078: Status 404 returned error can't find the container with id f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078 Oct 30 04:31:54.980892 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:54.980860 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" event={"ID":"4661b0bf-74a1-4221-a09f-7c74ba2efef7","Type":"ContainerStarted","Data":"f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078"} Oct 30 04:31:55.988839 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:55.988797 4560 generic.go:356] "Generic (PLEG): container finished" podID="4661b0bf-74a1-4221-a09f-7c74ba2efef7" containerID="a254454cfbd520890da85e6b3a4f8745acd1d44d78ce079ebd8773a3b7bc9954" exitCode=0 Oct 30 04:31:55.989349 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:55.988873 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" event={"ID":"4661b0bf-74a1-4221-a09f-7c74ba2efef7","Type":"ContainerDied","Data":"a254454cfbd520890da85e6b3a4f8745acd1d44d78ce079ebd8773a3b7bc9954"} Oct 30 04:31:57.413157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.413094 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:57.441798 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.441765 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\") pod \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " Oct 30 04:31:57.442014 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.441897 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\" (UniqueName: \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\") pod \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " Oct 30 04:31:57.442014 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.441967 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\") pod \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\" (UID: \"4661b0bf-74a1-4221-a09f-7c74ba2efef7\") " Oct 30 04:31:57.442160 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.442137 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca" pod "4661b0bf-74a1-4221-a09f-7c74ba2efef7" (UID: "4661b0bf-74a1-4221-a09f-7c74ba2efef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:57.442334 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.442307 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:57.444000 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.443975 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca" pod "4661b0bf-74a1-4221-a09f-7c74ba2efef7" (UID: "4661b0bf-74a1-4221-a09f-7c74ba2efef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:57.444125 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.444109 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb" pod "4661b0bf-74a1-4221-a09f-7c74ba2efef7" (UID: "4661b0bf-74a1-4221-a09f-7c74ba2efef7"). InnerVolumeSpecName "kube-api-access-9klxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:57.543178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.543131 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9klxb\" (UniqueName: \"kubernetes.io/projected/4661b0bf-74a1-4221-a09f-7c74ba2efef7-kube-api-access-9klxb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:57.543178 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:57.543167 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4661b0bf-74a1-4221-a09f-7c74ba2efef7-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:31:58.026015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:58.025975 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" event={"ID":"4661b0bf-74a1-4221-a09f-7c74ba2efef7","Type":"ContainerDied","Data":"f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078"} Oct 30 04:31:58.026015 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:58.026004 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74b8af6-ckc72" Oct 30 04:31:58.026306 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:31:58.026056 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ef0695e39f504329781f09f0fbcee8df6b5be6164e415b78f2ab33a0341078" Oct 30 04:32:04.537299 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.537221 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x"] Oct 30 04:32:04.537726 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.537709 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4661b0bf-74a1-4221-a09f-7c74ba2efef7" containerName="revoke-keys" Oct 30 04:32:04.537774 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.537727 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4661b0bf-74a1-4221-a09f-7c74ba2efef7" containerName="revoke-keys" Oct 30 04:32:04.537836 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.537827 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4661b0bf-74a1-4221-a09f-7c74ba2efef7" containerName="revoke-keys" Oct 30 04:32:04.541201 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.541179 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.566576 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.566541 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:04.569349 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.569319 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x"] Oct 30 04:32:04.579280 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.579248 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:04.579429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.579304 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:04.579429 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.579321 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:04.612413 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.612372 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.612643 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.612427 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2z7\" (UniqueName: \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.612643 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.612548 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.714040 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.714001 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2z7\" (UniqueName: \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.714040 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.714049 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.714304 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.714147 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.714787 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.714761 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.717045 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.717015 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.734340 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.734309 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2z7\" (UniqueName: \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\") pod \"maas-api-revoke-sub-bb59c1b4-dt76x\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:04.854825 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:04.854733 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:05.156448 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:05.156416 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x"] Oct 30 04:32:05.157432 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:05.157407 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbdff39_9bbd_4047_84a2_119ddd36fd9a.slice/crio-a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498 WatchSource:0}: Error finding container a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498: Status 404 returned error can't find the container with id a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498 Oct 30 04:32:06.074854 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:06.074818 4560 generic.go:356] "Generic (PLEG): container finished" podID="6bbdff39-9bbd-4047-84a2-119ddd36fd9a" containerID="95515db534bd124786855e8c38318e3122a3a8b15badd5b05d6a9d1a1d05210b" exitCode=0 Oct 30 04:32:06.075290 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:06.074905 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" event={"ID":"6bbdff39-9bbd-4047-84a2-119ddd36fd9a","Type":"ContainerDied","Data":"95515db534bd124786855e8c38318e3122a3a8b15badd5b05d6a9d1a1d05210b"} Oct 30 04:32:06.075290 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:06.074976 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" event={"ID":"6bbdff39-9bbd-4047-84a2-119ddd36fd9a","Type":"ContainerStarted","Data":"a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498"} Oct 30 04:32:07.489836 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.489795 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:07.543201 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.543166 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\") pod \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " Oct 30 04:32:07.543412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.543229 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\") pod \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " Oct 30 04:32:07.543412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.543267 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\" (UniqueName: \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\") pod \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\" (UID: \"6bbdff39-9bbd-4047-84a2-119ddd36fd9a\") " Oct 30 04:32:07.543680 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.543649 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca" pod "6bbdff39-9bbd-4047-84a2-119ddd36fd9a" (UID: "6bbdff39-9bbd-4047-84a2-119ddd36fd9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:07.543788 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.543722 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.545407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.545374 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7" pod "6bbdff39-9bbd-4047-84a2-119ddd36fd9a" (UID: "6bbdff39-9bbd-4047-84a2-119ddd36fd9a"). InnerVolumeSpecName "kube-api-access-mb2z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:07.546489 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.546467 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca" pod "6bbdff39-9bbd-4047-84a2-119ddd36fd9a" (UID: "6bbdff39-9bbd-4047-84a2-119ddd36fd9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:07.644795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.644758 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2z7\" (UniqueName: \"kubernetes.io/projected/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-kube-api-access-mb2z7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.644795 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:07.644790 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbdff39-9bbd-4047-84a2-119ddd36fd9a-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:08.125895 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:08.125864 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" Oct 30 04:32:08.126253 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:08.125889 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb59c1b4-dt76x" event={"ID":"6bbdff39-9bbd-4047-84a2-119ddd36fd9a","Type":"ContainerDied","Data":"a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498"} Oct 30 04:32:08.126488 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:08.126466 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b6c4f4613667c47eaad81a8512bc16e49090c9bdeea910ad95b7571d96c498" Oct 30 04:32:11.749398 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:32:11.749362 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:32:14.867592 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.867549 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72"] Oct 30 04:32:14.868007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.867967 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6bbdff39-9bbd-4047-84a2-119ddd36fd9a" containerName="revoke-keys" Oct 30 04:32:14.868007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.867979 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbdff39-9bbd-4047-84a2-119ddd36fd9a" containerName="revoke-keys" Oct 30 04:32:14.868093 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.868088 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbdff39-9bbd-4047-84a2-119ddd36fd9a" containerName="revoke-keys" Oct 30 04:32:14.871360 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.871343 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:14.900164 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.900117 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:14.900164 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.900122 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:14.900164 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.900126 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:14.900771 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.900177 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:14.902721 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.902697 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72"] Oct 30 04:32:14.918915 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.918880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2x42\" (UniqueName: \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:14.919073 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.918930 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:14.919140 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:14.919109 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.020071 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.020030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.020262 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.020130 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2x42\" (UniqueName: \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.020262 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.020150 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.020742 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.020721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.022414 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.022395 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.039775 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.039742 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2x42\" (UniqueName: \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\") pod \"maas-api-revoke-sub-4e1c265c-2pl72\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.184492 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.184407 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:15.484478 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:15.484447 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72"] Oct 30 04:32:15.486920 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:15.486892 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe92bfe2_0301_4756_bd6f_71d61889f162.slice/crio-0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7 WatchSource:0}: Error finding container 0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7: Status 404 returned error can't find the container with id 0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7 Oct 30 04:32:16.173161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:16.173070 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" event={"ID":"be92bfe2-0301-4756-bd6f-71d61889f162","Type":"ContainerStarted","Data":"e996f1c21c2014a869e40078abac3f6323241f4906680e1683aa77cb015b0731"} Oct 30 04:32:16.173161 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:16.173128 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" event={"ID":"be92bfe2-0301-4756-bd6f-71d61889f162","Type":"ContainerStarted","Data":"0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7"} Oct 30 04:32:16.214409 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:16.214361 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" podStartSLOduration=2.214345948 podStartE2EDuration="2.214345948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:32:16.212489221 +0000 UTC m=+5364.197609110" watchObservedRunningTime="2026-10-30 04:32:16.214345948 +0000 UTC m=+5364.199465853" Oct 30 04:32:17.193611 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:17.193571 4560 generic.go:356] "Generic (PLEG): container finished" podID="be92bfe2-0301-4756-bd6f-71d61889f162" containerID="e996f1c21c2014a869e40078abac3f6323241f4906680e1683aa77cb015b0731" exitCode=0 Oct 30 04:32:17.194333 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:17.194303 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" event={"ID":"be92bfe2-0301-4756-bd6f-71d61889f162","Type":"ContainerDied","Data":"e996f1c21c2014a869e40078abac3f6323241f4906680e1683aa77cb015b0731"} Oct 30 04:32:18.591064 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.590713 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:18.601550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.600636 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:32:18.601550 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.600703 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zkqzg" podUID="bdecf353-c6f0-4457-bdd4-98a4722c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:32:18.730344 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.730309 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\") pod \"be92bfe2-0301-4756-bd6f-71d61889f162\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " Oct 30 04:32:18.730594 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.730395 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\" (UniqueName: \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\") pod \"be92bfe2-0301-4756-bd6f-71d61889f162\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " Oct 30 04:32:18.730594 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.730474 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\") pod \"be92bfe2-0301-4756-bd6f-71d61889f162\" (UID: \"be92bfe2-0301-4756-bd6f-71d61889f162\") " Oct 30 04:32:18.730794 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.730767 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca" pod "be92bfe2-0301-4756-bd6f-71d61889f162" (UID: "be92bfe2-0301-4756-bd6f-71d61889f162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:18.730943 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.730925 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:18.732732 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.732703 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42" pod "be92bfe2-0301-4756-bd6f-71d61889f162" (UID: "be92bfe2-0301-4756-bd6f-71d61889f162"). InnerVolumeSpecName "kube-api-access-m2x42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:18.732895 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.732724 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca" pod "be92bfe2-0301-4756-bd6f-71d61889f162" (UID: "be92bfe2-0301-4756-bd6f-71d61889f162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:18.832693 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.832600 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be92bfe2-0301-4756-bd6f-71d61889f162-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:18.832693 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:18.832647 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2x42\" (UniqueName: \"kubernetes.io/projected/be92bfe2-0301-4756-bd6f-71d61889f162-kube-api-access-m2x42\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:19.229671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:19.229633 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" event={"ID":"be92bfe2-0301-4756-bd6f-71d61889f162","Type":"ContainerDied","Data":"0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7"} Oct 30 04:32:19.229671 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:19.229663 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1c265c-2pl72" Oct 30 04:32:19.229933 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:19.229684 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d670b6a8e2a3d5eb08414b69d860e36a7d198259c2ee29be62ed540ae2779b7" Oct 30 04:32:25.186418 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.186380 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4"] Oct 30 04:32:25.187033 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.187010 4560 cpu_manager.go:414] "containerMap: removing container" podUID="be92bfe2-0301-4756-bd6f-71d61889f162" containerName="revoke-keys" Oct 30 04:32:25.187129 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.187035 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be92bfe2-0301-4756-bd6f-71d61889f162" containerName="revoke-keys" Oct 30 04:32:25.187190 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.187169 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="be92bfe2-0301-4756-bd6f-71d61889f162" containerName="revoke-keys" Oct 30 04:32:25.190551 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.190509 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.209670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.209640 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4"] Oct 30 04:32:25.218886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.218829 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:25.218886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.218838 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:25.218886 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.218865 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:25.219159 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.218919 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:25.291691 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.291653 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.291871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.291782 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfvq\" (UniqueName: \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.291871 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.291817 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.393293 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.393248 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfvq\" (UniqueName: \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.393293 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.393299 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.393586 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.393399 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.394011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.393983 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.395763 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.395743 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.413195 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.413163 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfvq\" (UniqueName: \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\") pod \"maas-api-revoke-sub-e7b1e25c-mchz4\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.504846 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.504810 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:25.805254 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:25.805222 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4"] Oct 30 04:32:25.806155 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:25.806125 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28607bfa_9f29_4bf0_9ecf_b2327cb58181.slice/crio-dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24 WatchSource:0}: Error finding container dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24: Status 404 returned error can't find the container with id dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24 Oct 30 04:32:26.266241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:26.266205 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" event={"ID":"28607bfa-9f29-4bf0-9ecf-b2327cb58181","Type":"ContainerStarted","Data":"80ace578498eac33d6a6a508131bab0cd86a891e6948d7681f2fff5354f1191f"} Oct 30 04:32:26.266735 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:26.266253 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" event={"ID":"28607bfa-9f29-4bf0-9ecf-b2327cb58181","Type":"ContainerStarted","Data":"dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24"} Oct 30 04:32:26.306858 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:26.306809 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" podStartSLOduration=1.306795524 podStartE2EDuration="1.306795524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:32:26.304793124 +0000 UTC m=+5374.289913014" watchObservedRunningTime="2026-10-30 04:32:26.306795524 +0000 UTC m=+5374.291915442" Oct 30 04:32:27.275872 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:27.275823 4560 generic.go:356] "Generic (PLEG): container finished" podID="28607bfa-9f29-4bf0-9ecf-b2327cb58181" containerID="80ace578498eac33d6a6a508131bab0cd86a891e6948d7681f2fff5354f1191f" exitCode=0 Oct 30 04:32:27.276296 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:27.275898 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" event={"ID":"28607bfa-9f29-4bf0-9ecf-b2327cb58181","Type":"ContainerDied","Data":"80ace578498eac33d6a6a508131bab0cd86a891e6948d7681f2fff5354f1191f"} Oct 30 04:32:28.708571 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.690998 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:28.735382 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.735348 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\" (UniqueName: \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\") pod \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " Oct 30 04:32:28.735595 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.735496 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\") pod \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " Oct 30 04:32:28.735595 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.735563 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\") pod \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\" (UID: \"28607bfa-9f29-4bf0-9ecf-b2327cb58181\") " Oct 30 04:32:28.736100 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.736076 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca" pod "28607bfa-9f29-4bf0-9ecf-b2327cb58181" (UID: "28607bfa-9f29-4bf0-9ecf-b2327cb58181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:28.737888 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.737697 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq" pod "28607bfa-9f29-4bf0-9ecf-b2327cb58181" (UID: "28607bfa-9f29-4bf0-9ecf-b2327cb58181"). InnerVolumeSpecName "kube-api-access-zmfvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:28.738121 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.738097 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca" pod "28607bfa-9f29-4bf0-9ecf-b2327cb58181" (UID: "28607bfa-9f29-4bf0-9ecf-b2327cb58181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:28.836983 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.836949 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfvq\" (UniqueName: \"kubernetes.io/projected/28607bfa-9f29-4bf0-9ecf-b2327cb58181-kube-api-access-zmfvq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.836983 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.836979 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.836983 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:28.836989 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28607bfa-9f29-4bf0-9ecf-b2327cb58181-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:29.305267 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:29.305235 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" Oct 30 04:32:29.305460 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:29.305268 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b1e25c-mchz4" event={"ID":"28607bfa-9f29-4bf0-9ecf-b2327cb58181","Type":"ContainerDied","Data":"dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24"} Oct 30 04:32:29.305460 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:29.305328 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2d6cf9d82ab7f72bc5143e24464cdfb1c9b10d621d0aa7dfc05f4cb7441f24" Oct 30 04:32:35.525275 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.525234 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk"] Oct 30 04:32:35.525711 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.525696 4560 cpu_manager.go:414] "containerMap: removing container" podUID="28607bfa-9f29-4bf0-9ecf-b2327cb58181" containerName="revoke-keys" Oct 30 04:32:35.525769 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.525712 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28607bfa-9f29-4bf0-9ecf-b2327cb58181" containerName="revoke-keys" Oct 30 04:32:35.525807 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.525789 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="28607bfa-9f29-4bf0-9ecf-b2327cb58181" containerName="revoke-keys" Oct 30 04:32:35.529246 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.529227 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.548322 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.548280 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk"] Oct 30 04:32:35.557922 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.557857 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:35.557922 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.557865 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:35.558158 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.557945 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:35.558158 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.557959 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:35.603654 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.603613 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9n9c\" (UniqueName: \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.603846 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.603733 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.603846 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.603769 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.704570 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.704503 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.704784 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.704593 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.704784 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.704705 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9n9c\" (UniqueName: \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.705371 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.705345 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.707032 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.707009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.725422 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.725393 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9n9c\" (UniqueName: \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\") pod \"maas-api-revoke-sub-42bfc99f-gmjhk\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:35.843001 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:35.842909 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:36.149729 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:36.149699 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31523ef4_bf78_446e_83da_005b2b05ad77.slice/crio-18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58 WatchSource:0}: Error finding container 18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58: Status 404 returned error can't find the container with id 18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58 Oct 30 04:32:36.152400 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:36.152366 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk"] Oct 30 04:32:36.343056 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:36.343019 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" event={"ID":"31523ef4-bf78-446e-83da-005b2b05ad77","Type":"ContainerStarted","Data":"18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58"} Oct 30 04:32:37.352259 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:37.352164 4560 generic.go:356] "Generic (PLEG): container finished" podID="31523ef4-bf78-446e-83da-005b2b05ad77" containerID="e30ea4fc54cd366dd5ff442ddb1761217793506fd0e2c5c6eada3d294a928806" exitCode=0 Oct 30 04:32:37.352259 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:37.352234 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" event={"ID":"31523ef4-bf78-446e-83da-005b2b05ad77","Type":"ContainerDied","Data":"e30ea4fc54cd366dd5ff442ddb1761217793506fd0e2c5c6eada3d294a928806"} Oct 30 04:32:38.720675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.720644 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:38.737587 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.737552 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\") pod \"31523ef4-bf78-446e-83da-005b2b05ad77\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " Oct 30 04:32:38.737772 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.737617 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\" (UniqueName: \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\") pod \"31523ef4-bf78-446e-83da-005b2b05ad77\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " Oct 30 04:32:38.737833 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.737772 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\") pod \"31523ef4-bf78-446e-83da-005b2b05ad77\" (UID: \"31523ef4-bf78-446e-83da-005b2b05ad77\") " Oct 30 04:32:38.737931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.737905 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca" pod "31523ef4-bf78-446e-83da-005b2b05ad77" (UID: "31523ef4-bf78-446e-83da-005b2b05ad77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:38.738120 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.738102 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:38.739734 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.739712 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c" pod "31523ef4-bf78-446e-83da-005b2b05ad77" (UID: "31523ef4-bf78-446e-83da-005b2b05ad77"). InnerVolumeSpecName "kube-api-access-t9n9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:38.739904 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.739886 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca" pod "31523ef4-bf78-446e-83da-005b2b05ad77" (UID: "31523ef4-bf78-446e-83da-005b2b05ad77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:38.841172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.841129 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9n9c\" (UniqueName: \"kubernetes.io/projected/31523ef4-bf78-446e-83da-005b2b05ad77-kube-api-access-t9n9c\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:38.841172 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:38.841174 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31523ef4-bf78-446e-83da-005b2b05ad77-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:39.378674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:39.378638 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" Oct 30 04:32:39.378881 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:39.378702 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42bfc99f-gmjhk" event={"ID":"31523ef4-bf78-446e-83da-005b2b05ad77","Type":"ContainerDied","Data":"18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58"} Oct 30 04:32:39.378881 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:39.378759 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a0526c3b0a2c95e841cb716ecc8c54eab7d73793052b03f463fccc81237c58" Oct 30 04:32:41.754106 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:32:41.754070 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:32:45.910312 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.910272 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w"] Oct 30 04:32:45.910940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.910918 4560 cpu_manager.go:414] "containerMap: removing container" podUID="31523ef4-bf78-446e-83da-005b2b05ad77" containerName="revoke-keys" Oct 30 04:32:45.910986 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.910943 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31523ef4-bf78-446e-83da-005b2b05ad77" containerName="revoke-keys" Oct 30 04:32:45.911099 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.911087 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="31523ef4-bf78-446e-83da-005b2b05ad77" containerName="revoke-keys" Oct 30 04:32:45.923602 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.923564 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:45.932309 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.932275 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w"] Oct 30 04:32:45.941675 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.941643 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:45.941866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.941655 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:45.941866 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.941656 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:45.942007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:45.941732 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:46.025272 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.025234 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.025453 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.025301 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.025453 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.025426 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwfk\" (UniqueName: \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.126585 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.126546 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.126765 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.126654 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwfk\" (UniqueName: \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.126765 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.126707 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.127349 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.127326 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.128950 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.128922 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.146578 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.146547 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwfk\" (UniqueName: \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\") pod \"maas-api-revoke-sub-cc2cd826-s4l2w\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.240698 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.240657 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:46.540981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:46.540945 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w"] Oct 30 04:32:46.541829 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:46.541802 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c63c883_f1b8_4241_99fd_12d7f28e2967.slice/crio-f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028 WatchSource:0}: Error finding container f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028: Status 404 returned error can't find the container with id f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028 Oct 30 04:32:47.423425 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:47.423384 4560 generic.go:356] "Generic (PLEG): container finished" podID="2c63c883-f1b8-4241-99fd-12d7f28e2967" containerID="eab12fca454b7a998e3982dac384e0db7ddef214c4ca348d8d04293a2d2bb4c5" exitCode=0 Oct 30 04:32:47.423873 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:47.423466 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" event={"ID":"2c63c883-f1b8-4241-99fd-12d7f28e2967","Type":"ContainerDied","Data":"eab12fca454b7a998e3982dac384e0db7ddef214c4ca348d8d04293a2d2bb4c5"} Oct 30 04:32:47.423873 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:47.423540 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" event={"ID":"2c63c883-f1b8-4241-99fd-12d7f28e2967","Type":"ContainerStarted","Data":"f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028"} Oct 30 04:32:48.745967 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.745940 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:48.831981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.831061 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\" (UniqueName: \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\") pod \"2c63c883-f1b8-4241-99fd-12d7f28e2967\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " Oct 30 04:32:48.831981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.831212 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\") pod \"2c63c883-f1b8-4241-99fd-12d7f28e2967\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " Oct 30 04:32:48.831981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.831295 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\") pod \"2c63c883-f1b8-4241-99fd-12d7f28e2967\" (UID: \"2c63c883-f1b8-4241-99fd-12d7f28e2967\") " Oct 30 04:32:48.831981 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.831846 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca" pod "2c63c883-f1b8-4241-99fd-12d7f28e2967" (UID: "2c63c883-f1b8-4241-99fd-12d7f28e2967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:48.834174 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.834139 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk" pod "2c63c883-f1b8-4241-99fd-12d7f28e2967" (UID: "2c63c883-f1b8-4241-99fd-12d7f28e2967"). InnerVolumeSpecName "kube-api-access-vqwfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:48.834288 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.834203 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca" pod "2c63c883-f1b8-4241-99fd-12d7f28e2967" (UID: "2c63c883-f1b8-4241-99fd-12d7f28e2967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:48.932407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.932363 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwfk\" (UniqueName: \"kubernetes.io/projected/2c63c883-f1b8-4241-99fd-12d7f28e2967-kube-api-access-vqwfk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.932407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.932400 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.932407 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:48.932409 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c63c883-f1b8-4241-99fd-12d7f28e2967-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:32:49.437080 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:49.437047 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" Oct 30 04:32:49.437282 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:49.437084 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc2cd826-s4l2w" event={"ID":"2c63c883-f1b8-4241-99fd-12d7f28e2967","Type":"ContainerDied","Data":"f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028"} Oct 30 04:32:49.437282 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:49.437134 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4781f10601083b4c7f01184d8636d92c58687426fc278266b34319bf6920028" Oct 30 04:32:56.234837 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.234802 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57"] Oct 30 04:32:56.235253 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.235238 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2c63c883-f1b8-4241-99fd-12d7f28e2967" containerName="revoke-keys" Oct 30 04:32:56.235298 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.235254 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c63c883-f1b8-4241-99fd-12d7f28e2967" containerName="revoke-keys" Oct 30 04:32:56.235346 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.235337 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c63c883-f1b8-4241-99fd-12d7f28e2967" containerName="revoke-keys" Oct 30 04:32:56.238812 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.238785 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.255076 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.255046 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:56.267962 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.267931 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:32:56.268118 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.268016 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:32:56.268118 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.268060 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:32:56.270923 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.270897 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57"] Oct 30 04:32:56.305415 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.305376 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.305596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.305480 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.305596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.305588 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dj8d\" (UniqueName: \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.406505 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.406469 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.406692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.406534 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.406692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.406594 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dj8d\" (UniqueName: \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.407267 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.407243 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.408890 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.408869 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.426546 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.426505 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dj8d\" (UniqueName: \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\") pod \"maas-api-revoke-sub-da8d8590-9sz57\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.556404 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.556321 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:32:56.855872 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:56.855837 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57"] Oct 30 04:32:56.856670 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:32:56.856636 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46348e9_25d8_4b7f_a81c_2e0ba56260b7.slice/crio-2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d WatchSource:0}: Error finding container 2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d: Status 404 returned error can't find the container with id 2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d Oct 30 04:32:57.482098 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:57.482063 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" event={"ID":"c46348e9-25d8-4b7f-a81c-2e0ba56260b7","Type":"ContainerStarted","Data":"97a2ce38f94bc6ac5796cdaac5f39bbb4d5d0194ed24884081b5c39cf211a57b"} Oct 30 04:32:57.482533 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:57.482119 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" event={"ID":"c46348e9-25d8-4b7f-a81c-2e0ba56260b7","Type":"ContainerStarted","Data":"2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d"} Oct 30 04:32:57.522367 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:57.522318 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" podStartSLOduration=1.5223048879999999 podStartE2EDuration="1.522304888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:32:57.520657198 +0000 UTC m=+5405.505777089" watchObservedRunningTime="2026-10-30 04:32:57.522304888 +0000 UTC m=+5405.507424776" Oct 30 04:32:58.493538 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:58.493485 4560 generic.go:356] "Generic (PLEG): container finished" podID="c46348e9-25d8-4b7f-a81c-2e0ba56260b7" containerID="97a2ce38f94bc6ac5796cdaac5f39bbb4d5d0194ed24884081b5c39cf211a57b" exitCode=0 Oct 30 04:32:58.494190 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:58.494161 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" event={"ID":"c46348e9-25d8-4b7f-a81c-2e0ba56260b7","Type":"ContainerDied","Data":"97a2ce38f94bc6ac5796cdaac5f39bbb4d5d0194ed24884081b5c39cf211a57b"} Oct 30 04:32:59.936539 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:32:59.936057 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:33:00.047345 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.047309 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\") pod \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " Oct 30 04:33:00.047567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.047375 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\") pod \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " Oct 30 04:33:00.047567 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.047400 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\" (UniqueName: \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\") pod \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\" (UID: \"c46348e9-25d8-4b7f-a81c-2e0ba56260b7\") " Oct 30 04:33:00.047783 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.047752 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca" pod "c46348e9-25d8-4b7f-a81c-2e0ba56260b7" (UID: "c46348e9-25d8-4b7f-a81c-2e0ba56260b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:00.050302 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.050266 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca" pod "c46348e9-25d8-4b7f-a81c-2e0ba56260b7" (UID: "c46348e9-25d8-4b7f-a81c-2e0ba56260b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:00.056069 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.056038 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d" pod "c46348e9-25d8-4b7f-a81c-2e0ba56260b7" (UID: "c46348e9-25d8-4b7f-a81c-2e0ba56260b7"). InnerVolumeSpecName "kube-api-access-5dj8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:00.148563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.148446 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:00.148563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.148487 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:00.148563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.148503 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dj8d\" (UniqueName: \"kubernetes.io/projected/c46348e9-25d8-4b7f-a81c-2e0ba56260b7-kube-api-access-5dj8d\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:00.518740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.518709 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" Oct 30 04:33:00.518740 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.518731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da8d8590-9sz57" event={"ID":"c46348e9-25d8-4b7f-a81c-2e0ba56260b7","Type":"ContainerDied","Data":"2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d"} Oct 30 04:33:00.518962 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:00.518778 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e68f236f9f512c208366fcfd073ad985808005614e471c5679985d1e684da3d" Oct 30 04:33:06.571018 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.570971 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x"] Oct 30 04:33:06.572011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.571985 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c46348e9-25d8-4b7f-a81c-2e0ba56260b7" containerName="revoke-keys" Oct 30 04:33:06.572011 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.572012 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46348e9-25d8-4b7f-a81c-2e0ba56260b7" containerName="revoke-keys" Oct 30 04:33:06.572259 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.572245 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46348e9-25d8-4b7f-a81c-2e0ba56260b7" containerName="revoke-keys" Oct 30 04:33:06.577142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.577114 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.590692 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.590660 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x"] Oct 30 04:33:06.599612 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.599583 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:33:06.599778 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.599641 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:33:06.605476 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.605445 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\"" Oct 30 04:33:06.605611 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.605499 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:33:06.613167 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.613141 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmln\" (UniqueName: \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.613266 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.613176 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.613266 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.613218 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.714791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.714754 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmln\" (UniqueName: \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.714791 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.714789 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.715080 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.714829 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.715477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.715443 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.717251 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.717228 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.734650 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.734618 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmln\" (UniqueName: \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\") pod \"maas-api-revoke-sub-1ac55ff6-4d42x\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:06.891857 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:06.891772 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:07.195980 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:33:07.195946 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e61192_d6ce_488a_ab9e_da8515106892.slice/crio-32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c WatchSource:0}: Error finding container 32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c: Status 404 returned error can't find the container with id 32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c Oct 30 04:33:07.197347 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:07.197315 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x"] Oct 30 04:33:07.557007 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:07.556967 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" event={"ID":"32e61192-d6ce-488a-ab9e-da8515106892","Type":"ContainerStarted","Data":"65355969d9cad5b3dec0bb9478161c7cf849aea9328c995d067de43ccb0944c9"} Oct 30 04:33:07.557233 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:07.557026 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" event={"ID":"32e61192-d6ce-488a-ab9e-da8515106892","Type":"ContainerStarted","Data":"32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c"} Oct 30 04:33:07.598180 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:07.598119 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" podStartSLOduration=1.598105881 podStartE2EDuration="1.598105881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:07.596660684 +0000 UTC m=+5415.581780572" watchObservedRunningTime="2026-10-30 04:33:07.598105881 +0000 UTC m=+5415.583225768" Oct 30 04:33:08.616897 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:08.616318 4560 generic.go:356] "Generic (PLEG): container finished" podID="32e61192-d6ce-488a-ab9e-da8515106892" containerID="65355969d9cad5b3dec0bb9478161c7cf849aea9328c995d067de43ccb0944c9" exitCode=0 Oct 30 04:33:08.616897 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:08.616406 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" event={"ID":"32e61192-d6ce-488a-ab9e-da8515106892","Type":"ContainerDied","Data":"65355969d9cad5b3dec0bb9478161c7cf849aea9328c995d067de43ccb0944c9"} Oct 30 04:33:09.761470 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.761441 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:09.849155 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.849118 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\") pod \"32e61192-d6ce-488a-ab9e-da8515106892\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " Oct 30 04:33:09.849330 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.849179 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\") pod \"32e61192-d6ce-488a-ab9e-da8515106892\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " Oct 30 04:33:09.849330 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.849299 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\" (UniqueName: \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\") pod \"32e61192-d6ce-488a-ab9e-da8515106892\" (UID: \"32e61192-d6ce-488a-ab9e-da8515106892\") " Oct 30 04:33:09.849541 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.849495 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca" pod "32e61192-d6ce-488a-ab9e-da8515106892" (UID: "32e61192-d6ce-488a-ab9e-da8515106892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:09.849725 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.849710 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-openshift-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:09.851241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.851221 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca" pod "32e61192-d6ce-488a-ab9e-da8515106892" (UID: "32e61192-d6ce-488a-ab9e-da8515106892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:09.851324 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.851298 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln" pod "32e61192-d6ce-488a-ab9e-da8515106892" (UID: "32e61192-d6ce-488a-ab9e-da8515106892"). InnerVolumeSpecName "kube-api-access-zqmln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:09.951157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.951076 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e61192-d6ce-488a-ab9e-da8515106892-maas-api-xks-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:09.951157 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:09.951108 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmln\" (UniqueName: \"kubernetes.io/projected/32e61192-d6ce-488a-ab9e-da8515106892-kube-api-access-zqmln\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\"" Oct 30 04:33:10.631141 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:10.631107 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" Oct 30 04:33:10.631361 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:10.631145 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac55ff6-4d42x" event={"ID":"32e61192-d6ce-488a-ab9e-da8515106892","Type":"ContainerDied","Data":"32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c"} Oct 30 04:33:10.631361 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:10.631192 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b8ab311d91fac1eb1e167baa45e37c5f5da9f0128b68b3f0c9811e3716c38c" Oct 30 04:33:11.749438 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:33:11.749401 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:33:39.519670 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:39.519586 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:33:39.726406 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:39.726369 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:33:39.950840 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:39.950760 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:33:40.492536 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:40.492498 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:33:41.748935 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:33:41.748902 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:33:42.358326 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:42.358288 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-07d50594-pprcb_28d73e27-fccd-4542-90f4-77f57f8afc11/revoke-keys/0.log" Oct 30 04:33:42.556563 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:42.556490 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1ac55ff6-4d42x_32e61192-d6ce-488a-ab9e-da8515106892/revoke-keys/0.log" Oct 30 04:33:42.749306 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:42.749274 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2860ee91-sbgwj_c283196a-fe8c-4367-8b9d-d1a23463adf8/revoke-keys/0.log" Oct 30 04:33:43.129170 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:43.129094 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-42bfc99f-gmjhk_31523ef4-bf78-446e-83da-005b2b05ad77/revoke-keys/0.log" Oct 30 04:33:43.319604 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:43.319575 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4aeb5da1-gjwtl_86a04f8c-f9f6-4577-b307-3bfc3b13b8cb/revoke-keys/0.log" Oct 30 04:33:43.515254 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:43.515227 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4e1c265c-2pl72_be92bfe2-0301-4756-bd6f-71d61889f162/revoke-keys/0.log" Oct 30 04:33:43.707922 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:43.707891 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-559dd0f5-4gvvg_6118fafe-5cb0-41c5-a89a-38d664068d7d/revoke-keys/0.log" Oct 30 04:33:43.889989 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:43.889918 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7c5d2b10-97b85_e9ae5600-04aa-4e60-a870-f4fb80547ecc/revoke-keys/0.log" Oct 30 04:33:44.085010 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:44.084968 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a4b0c39b-lz4nj_6109b196-ff73-49ff-9ed9-e1e92eb87f69/revoke-keys/0.log" Oct 30 04:33:44.261982 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:44.261946 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a840ec60-z4kz7_9bd22dcc-b40d-4eed-b711-17f37259bd85/revoke-keys/0.log" Oct 30 04:33:44.454699 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:44.454663 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-abc831e5-x5zdc_7edefc74-23f3-4ede-a5fd-74837106d9cc/revoke-keys/0.log" Oct 30 04:33:44.655301 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:44.655219 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bb59c1b4-dt76x_6bbdff39-9bbd-4047-84a2-119ddd36fd9a/revoke-keys/0.log" Oct 30 04:33:44.837729 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:44.837695 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cc2cd826-s4l2w_2c63c883-f1b8-4241-99fd-12d7f28e2967/revoke-keys/0.log" Oct 30 04:33:45.035700 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:45.035667 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d658b3e6-6j274_dd5dddf7-2de3-4cb1-958e-031a144f5d92/revoke-keys/0.log" Oct 30 04:33:45.228813 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:45.228776 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-da8d8590-9sz57_c46348e9-25d8-4b7f-a81c-2e0ba56260b7/revoke-keys/0.log" Oct 30 04:33:45.407291 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:45.407217 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e7b1e25c-mchz4_28607bfa-9f29-4bf0-9ecf-b2327cb58181/revoke-keys/0.log" Oct 30 04:33:45.600636 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:45.600609 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ea2dc8fe-jqg44_c285c2b5-839e-4395-b801-123c3a7e43b1/revoke-keys/0.log" Oct 30 04:33:45.798879 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:45.798850 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f74b8af6-ckc72_4661b0bf-74a1-4221-a09f-7c74ba2efef7/revoke-keys/0.log" Oct 30 04:33:49.027485 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:49.027443 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg"] Oct 30 04:33:49.040187 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:49.040148 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559dd0f5-4gvvg"] Oct 30 04:33:50.150987 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:50.150952 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6118fafe-5cb0-41c5-a89a-38d664068d7d" path="/var/lib/kubelet/pods/6118fafe-5cb0-41c5-a89a-38d664068d7d/volumes" Oct 30 04:33:50.873241 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:50.873199 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-hzz4f_5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c/kube-auth-proxy/0.log" Oct 30 04:33:51.078321 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:51.078272 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-88m6n_0806baf1-a58a-48df-8c59-aabafc920d93/istio-proxy/0.log" Oct 30 04:33:51.932730 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:51.932701 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lkn75_dcf485c0-995a-4d10-ac7f-dd565faf112b/istio-proxy/0.log" Oct 30 04:33:52.961022 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:52.960978 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zkqzg_bdecf353-c6f0-4457-bdd4-98a4722c6ca9/istio-proxy/0.log" Oct 30 04:33:59.028065 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:59.028027 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc"] Oct 30 04:33:59.041052 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:33:59.041017 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc831e5-x5zdc"] Oct 30 04:34:00.152050 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:00.152013 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edefc74-23f3-4ede-a5fd-74837106d9cc" path="/var/lib/kubelet/pods/7edefc74-23f3-4ede-a5fd-74837106d9cc/volumes" Oct 30 04:34:11.748017 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:34:11.747974 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:34:13.271082 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:13.271051 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sgjx4_f98b781c-c557-4ac9-a65b-96cf4902994a/global-pull-secret-syncer/0.log" Oct 30 04:34:14.060412 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:14.060382 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r95wv_706f817b-2517-45e0-8fb5-75cdb54bfd8c/konnectivity-agent/0.log" Oct 30 04:34:14.647905 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:14.647867 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-97.ec2.internal_b6ab5ae10a035469fb072566acf5cdc5/haproxy/0.log" Oct 30 04:34:14.985355 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:14.985324 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:15.086991 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:15.086938 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:15.684238 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:15.684198 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:15.827609 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:15.827578 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:19.041112 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:19.041080 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:19.146421 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:19.146384 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:19.748766 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:19.748730 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:19.939889 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:19.939860 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:23.530727 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:23.530695 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:23.641062 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:23.641025 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:24.029535 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.029489 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj"] Oct 30 04:34:24.042432 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.042401 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b0c39b-lz4nj"] Oct 30 04:34:24.151931 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.151871 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6109b196-ff73-49ff-9ed9-e1e92eb87f69" path="/var/lib/kubelet/pods/6109b196-ff73-49ff-9ed9-e1e92eb87f69/volumes" Oct 30 04:34:24.265607 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.265565 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:24.448088 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.447996 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:24.547780 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.547736 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd"] Oct 30 04:34:24.548431 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.548407 4560 cpu_manager.go:414] "containerMap: removing container" podUID="32e61192-d6ce-488a-ab9e-da8515106892" containerName="revoke-keys" Oct 30 04:34:24.548564 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.548433 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e61192-d6ce-488a-ab9e-da8515106892" containerName="revoke-keys" Oct 30 04:34:24.548564 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.548545 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e61192-d6ce-488a-ab9e-da8515106892" containerName="revoke-keys" Oct 30 04:34:24.552343 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.552321 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.572803 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.572772 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd"] Oct 30 04:34:24.583136 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.583109 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"kube-root-ca.crt\"" Oct 30 04:34:24.583252 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.583188 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s9nqc\"/\"default-dockercfg-kgslt\"" Oct 30 04:34:24.583252 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.583209 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"openshift-service-ca.crt\"" Oct 30 04:34:24.601430 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.601402 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-proc\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.601583 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.601470 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-podres\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.601636 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.601583 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-sys\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.601674 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.601636 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-lib-modules\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.601714 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.601698 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvpl\" (UniqueName: \"kubernetes.io/projected/2953eb65-d297-4188-b5b9-b6e569cf6567-kube-api-access-sfvpl\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702186 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702153 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-podres\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702186 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702190 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-sys\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702208 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-lib-modules\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702252 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvpl\" (UniqueName: \"kubernetes.io/projected/2953eb65-d297-4188-b5b9-b6e569cf6567-kube-api-access-sfvpl\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702283 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-sys\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-proc\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702337 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-proc\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702343 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-podres\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.702454 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.702372 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2953eb65-d297-4188-b5b9-b6e569cf6567-lib-modules\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.722343 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.722315 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvpl\" (UniqueName: \"kubernetes.io/projected/2953eb65-d297-4188-b5b9-b6e569cf6567-kube-api-access-sfvpl\") pod \"perf-node-gather-daemonset-db6gd\" (UID: \"2953eb65-d297-4188-b5b9-b6e569cf6567\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:24.865679 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:24.865627 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:25.165899 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:25.165870 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd"] Oct 30 04:34:25.166554 ip-10-0-136-97 kubenswrapper[4560]: W1030 04:34:25.166500 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2953eb65_d297_4188_b5b9_b6e569cf6567.slice/crio-5fb604558df1fa824bb83c30b46129fc6d0b1067a5334c908175a733c3da6e55 WatchSource:0}: Error finding container 5fb604558df1fa824bb83c30b46129fc6d0b1067a5334c908175a733c3da6e55: Status 404 returned error can't find the container with id 5fb604558df1fa824bb83c30b46129fc6d0b1067a5334c908175a733c3da6e55 Oct 30 04:34:25.337448 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:25.337418 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:25.466499 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:25.466468 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:25.878775 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:25.878696 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/alertmanager/0.log" Oct 30 04:34:26.091767 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.091731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" event={"ID":"2953eb65-d297-4188-b5b9-b6e569cf6567","Type":"ContainerStarted","Data":"5f7d96dde82fa2252278f0156683685fbbbd64a21c0245a8531a4c94d2fc1532"} Oct 30 04:34:26.091940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.091813 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" event={"ID":"2953eb65-d297-4188-b5b9-b6e569cf6567","Type":"ContainerStarted","Data":"5fb604558df1fa824bb83c30b46129fc6d0b1067a5334c908175a733c3da6e55"} Oct 30 04:34:26.091940 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.091857 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:26.123828 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.123791 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:26.133025 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.132936 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" podStartSLOduration=2.13292339 podStartE2EDuration="2.13292339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:26.131158699 +0000 UTC m=+5494.116278589" watchObservedRunningTime="2026-10-30 04:34:26.13292339 +0000 UTC m=+5494.118043280" Oct 30 04:34:26.476477 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.476445 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/config-reloader/0.log" Oct 30 04:34:26.598248 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.598216 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/kube-rbac-proxy-web/0.log" Oct 30 04:34:26.599482 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.599450 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:26.722383 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.722353 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/kube-rbac-proxy/0.log" Oct 30 04:34:26.845988 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.845912 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/kube-rbac-proxy-metric/0.log" Oct 30 04:34:26.969384 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:26.969351 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/prom-label-proxy/0.log" Oct 30 04:34:27.095923 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:27.095890 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8551ab49-f11b-4823-bf2f-7b3a3c574c02/init-config-reloader/0.log" Oct 30 04:34:27.377424 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:27.377391 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-djkjf_452439f5-083c-48b5-a919-0e5b1f696619/kube-state-metrics/0.log" Oct 30 04:34:27.514863 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:27.514830 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-djkjf_452439f5-083c-48b5-a919-0e5b1f696619/kube-rbac-proxy-main/0.log" Oct 30 04:34:27.640552 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:27.640459 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-djkjf_452439f5-083c-48b5-a919-0e5b1f696619/kube-rbac-proxy-self/0.log" Oct 30 04:34:28.389009 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:28.388980 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rscmd_8388f2bc-bc94-4436-a40b-9d83f051fbb7/node-exporter/0.log" Oct 30 04:34:28.510463 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:28.510430 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rscmd_8388f2bc-bc94-4436-a40b-9d83f051fbb7/kube-rbac-proxy/0.log" Oct 30 04:34:28.634937 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:28.634910 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rscmd_8388f2bc-bc94-4436-a40b-9d83f051fbb7/init-textfile/0.log" Oct 30 04:34:29.121553 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.121500 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zs8c4_f5409a86-c8b7-4b16-8e85-7d4b1b62f149/kube-rbac-proxy-main/0.log" Oct 30 04:34:29.245004 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.244976 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zs8c4_f5409a86-c8b7-4b16-8e85-7d4b1b62f149/kube-rbac-proxy-self/0.log" Oct 30 04:34:29.369387 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.369356 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zs8c4_f5409a86-c8b7-4b16-8e85-7d4b1b62f149/openshift-state-metrics/0.log" Oct 30 04:34:29.507149 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.507118 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/prometheus/0.log" Oct 30 04:34:29.520022 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.519989 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:29.645513 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.645475 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:29.645767 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.645657 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/config-reloader/0.log" Oct 30 04:34:29.946655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:29.946558 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/thanos-sidecar/0.log" Oct 30 04:34:30.984596 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:30.984562 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/kube-rbac-proxy-web/0.log" Oct 30 04:34:31.007482 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.007439 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:31.088682 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.088655 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-hzz4f_5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c/kube-auth-proxy/0.log" Oct 30 04:34:31.284104 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.284020 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-88m6n_0806baf1-a58a-48df-8c59-aabafc920d93/istio-proxy/0.log" Oct 30 04:34:31.311501 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.311471 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/kube-rbac-proxy/0.log" Oct 30 04:34:31.432601 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.432573 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:31.433510 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.433475 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/kube-rbac-proxy-thanos/0.log" Oct 30 04:34:31.540787 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.540711 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lkn75_dcf485c0-995a-4d10-ac7f-dd565faf112b/istio-proxy/0.log" Oct 30 04:34:31.554655 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.554604 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cf31f133-8a68-4de1-ad78-cf47f4159c32/init-config-reloader/0.log" Oct 30 04:34:31.863542 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.863446 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zkqzg_bdecf353-c6f0-4457-bdd4-98a4722c6ca9/istio-proxy/0.log" Oct 30 04:34:31.924851 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:31.924804 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-575jt_94aaa487-607b-489a-b81d-133f5f47baad/prometheus-operator-admission-webhook/0.log" Oct 30 04:34:32.055861 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.055830 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f65bfbcf-prgds_7ff79c21-06e7-472a-9401-f8cddd431bdd/telemeter-client/0.log" Oct 30 04:34:32.108115 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.108088 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-db6gd" Oct 30 04:34:32.185283 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.185212 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f65bfbcf-prgds_7ff79c21-06e7-472a-9401-f8cddd431bdd/reload/0.log" Oct 30 04:34:32.307909 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.307879 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f65bfbcf-prgds_7ff79c21-06e7-472a-9401-f8cddd431bdd/kube-rbac-proxy/0.log" Oct 30 04:34:32.438232 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.438162 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/thanos-query/0.log" Oct 30 04:34:32.559575 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.559540 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/kube-rbac-proxy-web/0.log" Oct 30 04:34:32.680461 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.680428 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/kube-rbac-proxy/0.log" Oct 30 04:34:32.808998 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.808966 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/prom-label-proxy/0.log" Oct 30 04:34:32.940380 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:32.940343 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/kube-rbac-proxy-rules/0.log" Oct 30 04:34:33.062452 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:33.062381 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8689bf44b8-ffjhb_54d72c2a-cfd0-4420-b58f-56a7437b1a78/kube-rbac-proxy-metrics/0.log" Oct 30 04:34:34.636561 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:34.636530 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9qzhs_92c02093-c650-44e2-a1b1-a8c41c98f0c2/manager/0.log" Oct 30 04:34:34.746967 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:34.746934 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bp7bm_f1609387-b7a1-4e32-afce-6e893fb62865/manager/0.log" Oct 30 04:34:35.772591 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:35.772553 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jk86r_504a6d3e-f83e-4c3e-92a7-0db908fcfe00/manager/0.log" Oct 30 04:34:36.177142 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:36.177067 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-r2cdp_11c3ee93-3d9d-4681-a38f-f745d04799ff/server/0.log" Oct 30 04:34:37.540814 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:37.540783 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-746cf556c-g2m9k_65b369dc-a5de-408f-941b-f19fa2124ebb/console/0.log" Oct 30 04:34:37.681113 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:37.681081 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-gsppl_70d4fafe-092f-46f4-9f21-3d924bd42122/download-server/0.log" Oct 30 04:34:41.752680 ip-10-0-136-97 kubenswrapper[4560]: E1030 04:34:41.752645 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bhsr" containerName="extractor" Oct 30 04:34:41.851527 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:41.851479 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fh4pc_fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1/dns/0.log" Oct 30 04:34:41.980487 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:41.980448 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fh4pc_fe7f1b08-8e5d-478f-be3b-27ac7fc3dfb1/kube-rbac-proxy/0.log" Oct 30 04:34:42.231471 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:42.231436 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fv2ns_cdf7caae-56b4-4610-878e-3dd5a3dda185/dns-node-resolver/0.log" Oct 30 04:34:42.739938 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:42.739895 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-hzz4f_5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c/kube-auth-proxy/0.log" Oct 30 04:34:42.807002 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:42.806969 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-88m6n_0806baf1-a58a-48df-8c59-aabafc920d93/istio-proxy/0.log" Oct 30 04:34:43.045165 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:43.045080 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lkn75_dcf485c0-995a-4d10-ac7f-dd565faf112b/istio-proxy/0.log" Oct 30 04:34:43.367022 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:43.366857 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zkqzg_bdecf353-c6f0-4457-bdd4-98a4722c6ca9/istio-proxy/0.log" Oct 30 04:34:43.614510 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:43.614480 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6xf9_34bad7eb-bfe1-43a0-9c79-6a48f6fb57ea/node-ca/0.log" Oct 30 04:34:44.066467 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:44.066437 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-hzz4f_5b80a5d5-f055-4ab6-a5f7-0ea879f9a95c/kube-auth-proxy/0.log" Oct 30 04:34:44.130334 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:44.130305 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-88m6n_0806baf1-a58a-48df-8c59-aabafc920d93/istio-proxy/0.log" Oct 30 04:34:44.366828 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:44.366743 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lkn75_dcf485c0-995a-4d10-ac7f-dd565faf112b/istio-proxy/0.log" Oct 30 04:34:44.678935 ip-10-0-136-97 kubenswrapper[4560]: I1030 04:34:44.678852 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zkqzg_bdecf353-c6f0-4457-bdd4-98a4722c6ca9/istio-proxy/0.log"